Critically Evaluating Scientific Claims

← Navigator Home Page

Critically Evaluating Scientific Claims

Section 1 of 4

1 Introduction
2 Deconstructing a Scientific Claim
3 Interrogating Numbers and Statistics in Claims
4 Spotting Pseudoscience

The press release says “groundbreaking,” but is it?

A politician is calling something “sound science,” but … is it?

When incorporating scientific evidence into your reporting, it’s crucial to assess the credibility and context surrounding the claims being made. By asking critical questions about the source and background of a claim, focusing on evaluating its origins, potential influences, and overall support, you can strengthen the accuracy and trustworthiness of your journalism.

Deconstructing a Scientific Claim

Critically evaluating scientific claims involves breaking them down into smaller parts: Who’s making the claim? What is the evidence backing that claim up? And who stands to benefit by making that claim?

By thoroughly examining the origins of a claim, being vigilant about potential influences, and understanding the scientific backing, you can provide your audience with well-informed and contextualized reporting on scientific topics.

1 /

Origin of a scientific claim

You might encounter a scientific claim through various avenues, including a press release, political speech, video, poster, reader tip, conference presentation, book or film.

Consider the nature of the source itself. Is it a primary research report or a secondary interpretation or review? Is it a promotional piece published by a company?

Reflect on potential motivations that might shape how the claim is being presented. For instance, claims originating from a public relations office may emphasize positive angles. Understanding the source helps you gauge the need for further investigation.

2 /

Are there any factors that might influence the objectivity of the claim?

Identify any potential conflicts of interest (COI), which can be personal or financial ties that could signal a biased claim.These include funding sources (e.g., industry-sponsored studies) or a researcher consulting for a company and making claims about its products. Personal relationships that could benefit from a claim being made also constitute COIs.

Recognize that the presence of a COI doesn’t automatically invalidate a claim, especially when knowledgeable experts in a field may have industry connections. However, prioritizing transparency and clearly disclosing any potential COIs in your reporting is crucial.

3 /

What kind of foundation supports this scientific claim?

Consider whether the claim comes from or is endorsed by acknowledged experts in the relevant field. Can you communicate directly with these experts to gain deeper insights?

Determine if the claim is linked to a peer-reviewed publication. Peer review is a critical process that helps ensure the rigor and validity of scientific findings.

Ask about the methodology used to arrive at the claim. Understanding the process can offer valuable context for its reliability.

4 /

How does this claim fit within the broader scientific conversation?

Explore whether there are alternative interpretations or arguments that challenge the claim. Consider who is presenting these counterarguments and the evidence they offer.

Is the claim representative of a fringe view within a field? Be mindful of false balance creeping into a story about a scientific claim. Avoid featuring a claim that’s not backed by evidence for the sake of covering “all sides.”

Understand that scientific understanding evolves. A new claim might represent a developing area of research where consensus has not yet been reached.

5 /

What’s the nature of the language used to present the claim?

Be alert to language that seems designed to provoke an emotional response, such as fear-mongering or hyperbole. Keywords to look out for include “breakthrough,” “cure,” and “proof.”

Consider whether the framing of the claim might inadvertently misrepresent its scientific basis or significance.

Avoid adopting an exaggerated tone in your reporting unless it is clearly supported by robust evidence and context.

Interrogating Numbers and Statistics in Claims

It’s important to maintain a critical lens when reporting on numbers and statistics presented in scientific claims—for example, air quality data presented by a government agency, or statistics on the effectiveness of a public health campaign, or a claim from a school district about jumps in reading scores after their school-based intervention.

Here are some pointers to help you harness the journalistic skills you already have as you evaluate claims:

Approach numbers and stats with a journalist's mindset

Claims with striking numbers can indicate newsworthiness and influence people's daily decisions. But it’s important to make sure those numbers stand on solid evidence. Think of evaluating numbers in claims like vetting a source. Just as you would investigate a source's background and motivations, apply the same critical thinking here.

Self-awareness is key. Confirmation bias can lead you to favor information that aligns with your beliefs. Check your initial reaction to a statistic, and make sure that doesn’t influence your reporting. (Pro tip: Using incognito mode for online research can help curb search engine bias, too).
Maintain healthy skepticism. Remember that corporate, government, and even academic public relations arms are inherently biased and may overstate claims.

Examine the evidence

With your built in B.S. detector, you already have the necessary tools to dig deeper and scrutinize the accuracy of stats within a claim. Don’t be intimidated by the numbers; seek underlying evidence as you would any source’s statement.

Do the numbers seem “too good to be true”? That’s a sign you should look closely at the evidence backing a claim.
Do you trust the track record of the person or institution sharing the statistic?
Is the statistic attached to a peer-reviewed publication or dataset you can review?
Sometimes, claims don’t align with the data behind them. Scientists might use a surrogate endpoint to measure an experiment’s effectiveness—for example, measuring blood sugar or cholesterol levels instead of the number of heart attacks a drug prevents. Understand what the numbers touted in a scientific claim truly represent.
Sometimes you might need to do a little extra math to grasp the significance of a statistic. For instance, if a press release purports a 100 percent increase in EV charging stations in a community, look for the original number. A jump from 1 to 2 is a 100 percent increase, just like 100 to 200, but the impact is different.

Check your understanding

Don’t hesitate to ask an outside expert (on or off the record) whether numbers in a claim seem legitimate and important in their field.

Spotting Pseudoscience

Claims about science come from all kinds of sources, whether it’s a medical diagnosis made in a hospital drama or the advertised healing properties of some handmade goods. We don’t have to assume bad intentions—or even that these claims aren’t true.

But illegitimate scientific claims can easily slip through the cracks and into stories—especially stories about other subjects, or those being drafted on a quick deadline. That’s why it’s important to keep an eye out for pseudoscience.

See if you can identify weak spots or pseudoscience in the claims below. Read the claim, then click the arrow to reveal the answer.

“This new construction plan will greatly benefit the environment."

Vague claims

If you see a sweeping, vague reference to a project or products’ benefits to “environment,” “health,” or another very broad scientific term, probe for more information. Is there more specific evidence behind the claim or is it based on buzzwords?

"Several local schools have reported a noticeable improvement in student engagement since they implemented the new ‘MindHelp' educational software, showing that it's revolutionizing learning."

Lack of evidence

If you’re struggling to find peer-reviewed studies, independent expert sources, or specific evidence to support a claim; or if the support for a claim primarily consists of anecdotal evidence or personal testimonials, you probably shouldn’t repeat it.

"Residents point to several other towns where property values dropped after an incinerator was built, proving that our property values will inevitably decline."

Confirmation bias

Are you noticing a lot of confirmation bias in the cited evidence around a claim? Selectively highlighting certain outcomes while ignoring others could be a warning sign that you’re dealing with some shady science.

"As everyone knows, drinking eight glasses of water a day is essential for optimal health."

Clichés or familiar notions

Debates over whether chocolate is good for you, for example, are always making headlines in popular media. Flippant statements about an ingredients’ health benefits, or a practice’s effect on the environment, should always perk your ears a bit—even if you’re pretty confident that they’re true. When we’ve heard something repeated a lot, it’s all too easy to implicitly accept it.

"Dr. Green, a local wellness advocate, says that electromagnetic frequencies from power lines are causing widespread health problems in our community."

Dubious expertise

If someone touts or implies a scientific background, but cursory research shows that they don’t have one, your skepticism is warranted.