Scientists often use statistical tests to measure effects in their studies. For example, did an Alzheimer’s drug improve participants’ recall ability? Or, did a reading intervention boost students’ test scores?
Don’t panic when you come across a series of statistical tests as you read scientific papers. Even if math wasn’t your favorite subject in school, understanding a few key concepts can go a long way towards interpreting the results of a study—and generating questions to ask your sources.
There are several statistical terms that come up frequently in scientific studies. Flip the cards
below to help you better understand what you’re looking at.
A relationship between two variables
Risk is often a central component of science stories. For example, what’s the risk of contracting bird flu after exposure? How likely are wildfires in your area during the summer months? What’s the probability that a public safety policy will mitigate the risk of gun violence in a community?
Covering risk with accuracy and nuance can be tricky—especially when studies report different kinds of risk or include odds instead of risk. Familiarizing yourself with the concepts below will help sharpen your reporting on risk.
Practice identifying key statistical concepts in reporting scenarios. By understanding these terms in context, you’ll strengthen your ability to interpret them for your audience.
With a few extra statistical tests or clicks in a data-analysis program, researchers can—intentionally or not–skew their results. Misleading statistics can crop up easily when researchers track many different variables as part of any one study—a common practice in science.
For example, a psychologist interested in the relationship between implicit gender bias and employment discrimination might track hiring managers’ scores on perceptions of candidates’ competence, time spent reading cover letters, whether they invite an applicant for an interview, and many other behaviors.
When analyzing their results, however, researchers have to make thoughtful decisions about which combinations of variables to test and how to test them appropriately. Look for the questionable practices below as you comb through a study’s results section. Spotting a red flag doesn’t automatically mean a researcher has done something unethical or even incorrect, but it should prompt a question in the interview.
P-hacking is the practice of “mining” data for significant results.
This can involve researchers running a large number of statistical tests until they obtain the statistically significant result they prefer. The more statistical tests they run, the greater the chance a statistically significant finding will pop out just by chance. Many fields follow the threshold of p < .05 as significant, meaning there’s a less than 5 percent chance that their finding occurred randomly. So, running 20 or more statistical tests means it’s pretty likely at least one will come back significant.
While trying out several different combinations of variables—what’s known as doing multiple comparisons —isn’t necessarily a dubious practice, a large number of tests could be a sign of p-hacking. If you spot a large number of tests run on the same data, ask the researchers about their methods to correct for this.
Researchers might run some preliminary tests on their data before they’re done with data collection. But “peeking” at data in this way, known as interim analysis, can artificially increase the chance of getting a significant finding. This is because it invites p-hacking, and it could prompt a researcher to swap out a study’s endpoints or alter an intervention in progress.
If the methods of a paper suggest that researchers analyzed some data early, ask them to explain why and how they controlled for multiple comparisons.
Sometimes, researchers measure outcomes that are far removed from truly meaningful real-world consequences. Behavioral researchers might measure teens’ self-reported intention to use substances rather than measuring their actual substance use. And environmental researchers might measure concentrations of airborne particulates rather than measuring population health outcomes such as rates of asthma or premature death.
Such surrogate endpoints aren’t inherently flawed, but journalists should always ask researchers what a study’s favorable results actually mean for people’s daily lives.
As you work through the statistics reported in a study, remember you don’t have to go it alone. Rely on your sources’ expertise by asking them to vet your understanding and explain tricky concepts. There are also several free resources designed to help journalists sharpen their math skills and interpret weedy stats sections.
