Trust the science, cherry-picking the numbers, and shaping the narrative are phrases we've heard plenty of this year as government officials, and the media, predicted and forensically dissected election results, the 2020 Census, and the inescapable grip of updated COVID-19 data. We humans are incorrigible when it comes to skepticism, and, generally speaking, for good reason.
Methods of data collection are crucial to the process, but trusting the results involves deeper discussion. Inzata Analytics provides a workable primer on the pitfalls of data analysis: Who or what was sampled? Did researchers work from a pre-selected list, or did they contact people randomly? What type of person responds to market surveys? Are survey questions well designed? What margin of error is acceptable? Who's paying for the research, and to what extent is bias then likely?
The 1790 Census consisted of 650 federal marshals and their assistants who traveled the 13 states, door-to-door, and counted 3.9 million non-Native Americans, including 700,000 slaves; total cost: $44,000. This year, an estimated $15.6 billion has been allocated to count what's expected to be 335 million U.S. inhabitants, give or take. Will we trust the final numbers? Will we know that the follow ups reached those who are hard to reach? Did the questions on the Census forms clearly delineate demographic data? For 2030, some recommend rolling census tallies, forming a national register that includes overlapping accountability from federal, state, and local lists. Why does the data matter? Billions of dollars in grants, representation in the U.S. House, and a better sense of who we are as a nation, that’s why.
When you read about these numbers, or scour a chart from your favorite news source, what do you take away from that experience? Joy Eakins, President of Cornerstone Data, believes that we're deluged with data on a daily basis, but don't always approach the truth of the matter; that we may not receive the intended message, or worse, be fed the wrong data. For the next few months, we'll delve into various scenarios that affected our lives, and what we may have missed the first time we absorbed the information.
The Spike that Wasn’t
Joy Eakins
Hindsight may be 20/20 but learning from past mistakes is a noble pursuit. Take this summer’s reporting about COVID cases in Kansas. The White House Corona Virus Task Force issued a report on July 14 stating that Kansas was in the Red Zone – a virtual hotspot for cases – growing faster than the rest of the nation. According to the report, Kansas had a 150% increase in cases over the previous week.
Citizens and the media called on officials to do something. So, they did.
The Sedgwick County Health Officer issued a new order on July 21 that was later approved by Commissioners. Citing the White House report as a reason for the change, he closed bars and night clubs to the public and limited gatherings until Sept 9th. Six subsequent orders (Aug 18, Sept 3, 14, 17, Oct 16, 20) cited the July White House report as a reason for closing or limiting business operations and gatherings in Sedgwick County.
Wichita’s Mayor, Brandon Whipple tweeted multiple times about the “huge spike in numbers.” He told CNN in July that hospitals would be overwhelmed in just 14 to 21 days and in August tweeted that a “huge spike” was keeping kids out of schools and delaying the start of Fall activities.
Governor Kelly and her team responded, too. In perhaps one of her more controversial decisions, she issued an Executive Order to attempt to delay the start of all schools in the state.
Put yourself in their place – you just got word from the White House that your state is a hotspot for infections. It’s the White House. Surely they have the best analytics team in the nation preparing these reports. And not one single epidemiologist, analyst, reporter, or politician questioned the graphs or the data – until now.
If you read the White House report issued on July 14, you’ll see that in the State graph and in 7 of the county graphs, there was a “spike” in cases on the exact same day. It seemed unlikely that 7 of the 12 counties would have the same exact pattern as the state. So, we pulled the raw data cited by the White House and noted that there was a crucial flaw. Nine hundred thirty-two cases reported by KDHE on July 3 were included in the White House report on July 6. This created the appearance of a drop in cases one week followed by a major spike the following week.
Cornerstone Data analysis of USFacts data; Chart: Candor Visuals
While there was an increase in cases that week in July, the increase was only 13% - not the 150% reported widely in the news. In fact, while cases did increase over the summer, the week over week percent change in new cases trended down from June 27 to July 27 and the cases per 100,000 hovered right between 100 and 150 through September 25 before taking off in late October. The growth in case rates in Kansas was below the growth in the nation until the end of August.
What’s equally troubling is that the White House report and KDHE have used the “Report Date” as the axis for their charts. We’ve graphed it using the Symptom Onset date to show the actual growth of cases over time.
Cornerstone Data analysis of KDHE data; Chart: Candor Visuals
Since KDHE only reports the data three times a week the graphs appear to tell different stories to the casual user. For example, it looks like the Kansas case count was over 1,000 per day in the first chart but in reality, the first time 1,000 cases were observed in the state in a single day was October 5. In the first chart, it appears that cases rocketed up from about 280 per day in May to over 800 per day in June when in reality, that rate grew from late May to early October. Kansas is the only state that limits their reporting to 3 times a week, and it makes spotting trends in the data difficult and creates a misperception of spikes that aren’t always present.
The impact of mistakes and misperceptions like this have far-reaching implications and underscore the importance of letting the data inform decisions. According to analysis by Opportunity Insights, small business revenue is down 39% compared to January and owners have exhausted their savings and financial assistance with the series of closures in the early part of the pandemic - before the spike we saw in late October. With more than a quarter of small businesses now closed permanently, those remaining argue they must stay open to survive even if the case numbers are worse.
Additionally, school districts like Wichita kept children ages 11 and up at home in remote school. Many of our state’s children have not been in a school building for 9 months now. Student progress in math for low-income students is down almost 50% in Kansas and online math participation is down 54%.
The public’s good will has diminished as they have grown weary of official statements that never seem to pan out. Officials are accused of fear-mongering and blowing things out of proportion. And now we see public health officials and other leaders leaving their roles – perhaps at the moment we most need them.
Kansas is managing an emergency situation that will last months. We have limited resources and public will, so leaders need to deploy the appropriate measures at the right time in order to balance the harm of shutdowns with the impact of the virus. Our lack of data maturity means these leaders are guiding the efforts the way a captain would guide his ship through a storm without a motor – relying on his gut and experience and a glimpse at the stars through the clouds for direction. We’ve got to get the engine online if we want to minimize the damage and, most importantly, save the lives of the people we hold dear.