We like people to go “by the numbers.” Many believe that our ability to quantify everything is important. Test scores rank competitive outcomes for academic awards. The nightly news tells us about the stock market—to the decimal point. Sports analysis measures exact percentages about every statistic any person could ever need. If we can place numbers on a spreadsheet people seem more comfortable. In many senses, numbers run the world.
Ed Stetzer, president of LifeWay Research knows numbers. He works with them every day. For his part, Stetzer is concerned about how people use statistics in research. We all know that numbers can be tortured to give up the information we want to hear. Stetzer makes the point that “one conclusion from one study is no foundation for a theory on the future of a society.” He continues, “To get the whole picture, responsible researchers look at various studies, their methodologies, and their results. We must understand the parts in light of the whole. We should interpret each finding in light of the full study, and interpret each study in light of other studies. We reach bad conclusions when we latch onto one finding of one study, drag it out of context, and proclaim it from the rooftops without knowing whether our interpretation is justified.”[1]
Take, for example, the headline “Study Shows Canadians Healthier than Americans.” I sent this research on to my friend Doug who lives in Ontario, Canada. Being a doctoral student in the midst of doing his own research Doug pointed out the flaws in the study. Foundational to his comments was the simple fact that the research had been dependant upon phone interviews. “Imagine,” he wrote to me, “what my fellow Canadians thought when they were told the study was a comparison between themselves and Americans. Being better than America may have contributed psychologically to the answers.”
This vignette, which in itself proves nothing, simply points out a consistent contention about research—as soon as humans invest themselves personally in a project the project bears the bias of the researcher. With that in mind, I smiled when I read Jim Collin’s book Good to Great some time ago. In the introduction Collins wrote, “We developed all of the concepts in this book by making empirical deductions directly from the data.” Sounds good, right? But judging from the knock down, drag out verbal confrontations between the researchers peppered throughout the rest of Good to Great the problem of human interpretation casts a long shadow over the research process. As soon as researchers invest themselves in a study their viewpoint impacts the data.
If I only see through my eyes without proper controls, my data or choice of method in research could be dangerous. Ed Stetzer is right: being careful with numbers in research is important. The next time we read statistics perhaps we should ask, “Do the numbers tell me about the spirit of the athlete? Do test scores explain the success of those who did not graduate from college? Does polling explain the health habits of a nation?” If we do everything only by the numbers, our interpretation may be off by more than a decimal point. For Prime Time America, this is Dr. Mark Eckel, personally seeking truth wherever it’s found.
By the Numbers Moody Radio Commentary March, 2010
Dr. Mark Eckel, Professor of Old Testament, Crossroads Bible College
[1] Ed Stetzer, “Chicken Little was Wrong,” Christianity Today January 2010 (54:1), 36.