Science journalists are not just storytellers. They serve as watchdogs, too, informing readers about everything from the environmental impact of space delivery capsules to product launches of drugs that might end up in your medicine cabinet. Reporters on the science beat often spend time perusing papers, following researchers into the field, and talking to independent sources to present an accurate picture of scientific findings. When story leads slip into our inboxes in the form of press releases—about big companies, government agencies, or other institutions whose information is not easily available in a study’s methods section—what are the next steps? How do reporters sift through the PR claims and verify information?
Distinguishing industry hype from reality takes work. But the risks of not doing so are great. Even seasoned journalists wrote flattering profiles of the now-discredited diagnostics company Theranos’ founder Elizabeth Holmes and her company’s “revolutionary” blood-testing technology before a more critical, investigative reporter uncovered fraud. Similarly, two companies that generated media interest around video games for improving cognitive skills were fined by the Federal Trade Commission for false advertising. And the private space exploration company Mars One dazzled journalists and astronaut hopefuls alike with its aim of colonizing the Red Planet, until a series of articles called the mission’s feasibility and finances into question. Eventually, the company quietly declared bankruptcy in January 2019.
The process of vetting claims from companies often begins with a pitch from a media relations representative. And there’s an ever-growing cadre of press officers reaching out: Today, PR professionals outnumber journalists by about 6 to 1, compared with a 3-to-1 ratio a decade ago. As a result, journalists are inundated with pitches and must be increasingly careful about bias and exaggerated claims. I asked four science writers to discuss how they verify and seek outside perspective on industry PR claims. Below is an edited version of our discussion.
The journalists participating in the discussion were:
Nidhi Subbaraman, a science reporter at BuzzFeed News, who has previously covered science and technology at The Boston Globe and NBCNews.com.
Hal Hodson, The Economist’s Asia technology correspondent.
Sarah Scoles, a freelance journalist who is a contributing writer at Wired and a contributing editor at Popular Science. She covers astronomy, physics, and space technology.
Emily Hayes, a senior writer at the pharmaceutical industry publications Scrip and Pink Sheet with over 20 years of experience writing and editing articles about medical advances.
Knvul: How do you select an independent source to give context for industry news when there isn’t a published study to comment on?
Nidhi: Ideally, I’m on the hunt for someone who knows a lot about that industry, and is incentivized not to artificially talk it up. For emerging businesses like startups, venture capitalists who invest in other similar companies can offer a gut check on background, even if they will not comment on the record. (It can take a minute to build up that network—VCs tend to be harder to reach and less inclined to speak to reporters than researchers at universities.) For established areas, there are analysts on staff at banks and investment firms whose job it is to very closely watch their assigned sector, and they do talk to reporters.
Emily: I have developed sources over the years. If an area is new, I check [scientific] meeting programs for sources and I look for experts who have written review articles in peer-reviewed journals. I also will ask the PR office of the meeting to recommend independent expert sources.
Hal: It really does help if you know your beat well. Try to actually understand the stuff you’re writing about, rather than just relaying information. This will let you have in-depth conversations with sources about your story, rather than asking them to simply comment on material. It will also make your story better. If you don’t know a good person to talk to off-hand, throw the topic into Google Scholar and talk to the authors of the most-cited papers you can find. These conversations are one of the most enjoyable parts of my job.
Sarah: I usually start with someone to whom the discovery or technology would be useful—so, say, a potential customer. If a company that uses satellites to take pictures of Earth says, “Our data would be really useful to oil companies who want to watch how much oil arrives in a given port, or organizations interested in keeping track of deforestation,” I can call an oil company or a conservation organization, tell them about the data, and ask how they might use it.
I’ll also talk to competitors, both to get context on the news itself but also because it’s useful to readers to give a more complete picture of a given area. A satellite imaging company launched five instruments at once on Tuesday? They say they’re going to change the world? Cool, bro. But maybe, on that same rocket, another, very similar company sent up satellites that use a different technology to “see” a greater level of detail.
My favorite tool (weapon?), however, is using consultants. Consulting companies task people with taking broad and deep looks at a given area of research or innovation. They talk to the key players in that sector—and those players are, sometimes, more honest with the consultants than they will be with a journalist. A good consultant can have a handle on a whole field, in addition to details about individual elements. Plus, they’re often more used to talking to non-specialists, so no awkwardness or sentences made just of jargon.
Knvul: Is there a difference in how you evaluate industry press releases versus academic or government press releases?
Nidhi: I always treat press releases with caution!
Emily: I also treat all press releases pretty much the same.
Sarah: In general, I’m most skeptical of industry PR, because the people putting the news out there have more of a financial interest in getting a very particular story out. The statements that companies make—often cloaked in impenetrable business jargon that barely means anything at all—aim to advance a narrative that’s good for business, even if it’s misleading, and even if what’s good for business isn’t great for the rest of the world. I’ve had a lot of companies tell me they were “the first” to do something, and it hasn’t been true. I’ve had them point only to the heartwarming, humanitarian applications of their technology, when it’s mostly used for military applications.
And I’m (slightly) more skeptical of government press releases than I am of academic ones. Federal agencies and organizations have a long history of obscuring information they don’t want the public to know about, or slanting true information they do release to avoid upsetting people or making themselves look bad.
But of course, academic press releases have their own aims: They’re there to elevate the university, to elevate the scientist, to attract more funding, and—usually—to portray a given discovery as having resulted from an uncomplicated, straightforward path, as monumental, as novel, and as something whose headline you’d click on. In reality, science is iterative, nothing is truly novel, scientists are human, and money and prestige leak into all aspects of research. So in the end, I actually end up treating all three kinds of press release the same way: either as something to send to the trash can, or as a seed for a tangential story idea. For the most part, it’s not in a science journalist’s or the public’s best interest to write the same story that’s in the press release, because that’s probably the least interesting story, not the whole story, and not a contextualized story.
Hal: I rarely read press releases, but my default position is that people who are paid by Organization XYZ to present information about Organization XYZ’s products/services/work are not a great starting point for a story. If you’re writing about what PR puts in front of you, as it was presented to you, I think that’s a big mistake. I try and follow a rule that my attention as a reporter is always facing out into the world. This means I start working on stories because I find some interesting information while actively looking, not because some information is placed in my inbox. I find the stories, they don’t find me. I talk to comms people of course, but ideally, I am the one who initiates. I am the one framing the conversation.
Being able to ignore press releases is absolutely a privilege of the way my job at The Economist works. Not everyone will be able to do this. And to be fair, I probably do miss some stories because of it. But it’s become a point of principle for me. It feels like a very basic way of being true to that (somewhat twee) Orwell quote about journalism being what someone doesn’t want written.
Knvul: How should journalists verify statistical or other information offered in press releases?
Nidhi: Whenever possible, I try to find an unaffiliated group or organization that collects the same kind of information, but is not invested in the original source’s success (or failure!). Sometimes that’s a non-profit group, sometimes that’s a federal agency. If it’s a private company offering information about itself, such as growth figures or staff stats, outsiders may not have access to that data. In such instances, I would cite the company as the provider of that information, but offer a dab of context for how they may compare with others in the same space.
Hal: It can be really hard. Stats are the perfect tool for making self-validation matryoshka dolls that are hard to unpick. My best advice is to know your beat inside out, know it better than the folks who write the press releases. That will give you a general sense for when stuff is off. Apart from that you can’t do better than Carl Sagan’s tools for skeptical thinking: Confirm all the facts independently, think of every possible explanation, find out the relevant points of view, remember that scientific “authorities” have made mistakes too, and don’t get overly attached to any one hypothesis.
Emily: Journalists writing about health and biotech should take some time to understand the basics of clinical research and statistics. So the journalist can do his or her own analysis of information. They should look up other study results and review papers in the same disease area and see how the results compare. They can also look for editorials in the journals where the data were published and ask independent experts about the data.
Sarah: Numbers and other details have to come from somewhere. For scientific discoveries, that’s often an academic paper or conference presentation. For government sources, agencies often have semi-obscure technical-reports databases, where researchers may have—along the way, before they were at the point of writing a press release—produced white papers or status updates about their projects. NASA has the Technical Reports Server, the Pentagon has the Defense Technical Information Center, the Department of Energy has the Office of Science and Technical Information. You get the gist!
Companies, though, can be hard nuts to crack, as they have little (if any) public-reporting accountability, and much of their information is proprietary.
So for all of the above, it’s key to get confirmation, in an interview with the subject, of the claims, and to get more of the context around them. But you can also ask outside sources—competitors, colleagues—if they think the information sounds reasonable, or suspect. Also, like I mentioned, talk to consultants, whose job it is to hold a whole sector in their hands and make sense of it.
“Anything that sounds remotely hyperbolic: Firsts, bests, mosts, onlys. These milestones are easily tested with a call to an outside expert.”
Knvul: What are some red flags that make you suspicious of PR claims?
Nidhi: Anything that sounds remotely hyperbolic: Firsts, bests, mosts, onlys. These milestones are easily tested with a call to an outside expert.
Emily: For me, it’s when important aspects of the data are left out, such as p-values, or when the press release announces success based on results citing an improvement, but the study has actually failed to significantly improve the primary endpoint.
Hal: I think reporters should be suspicious of all PR claims. I’d be extra cautious about superlative claims, which are almost never true; graphs without non-zeroed x or y axes; the use of science-y words without citation of peer reviewed literature; off-record and background conversations with PR people.
I also think that virality itself is a red flag. Viral information has to be shallow and simple if it is to have the broad and quick appeal that makes something spread fast. The Boston Dynamics videos were a good example of this. The latest demo videos of their humanoid robots doing parkour were all over my timelines in October, and it just didn’t smell right. I tweeted out all the reasons I thought the vids were probably stage-managed. Nick Thompson asked the CEO about it on stage a couple of days later and he confirmed that they have to do more than 20 runs before the robot learns the course. It was clear to me that there was no way the robots could be as good as they seemed. If they were, we’d all have robot butlers already. The truth about what robots can and cannot do, and how they are programmed, is far more interesting and complex than what is presented in those videos, but it will never go viral because digesting the information requires more attention than is available through social media.
Sarah: Also, the more crafted and pointed a press release seems to be, the more suspicious I am. Of course, it’s in a PR office’s best interest to present a ready-made story with a compelling angle. They’re just doing a good job doing their job—because more outlets will pick up a tale like that. But an angle or a narrative, by definition, tells just one version of the truth. And if it’s the version the PR people want out there, you have to ask yourself, “But why this and not that or that?”
Knvul: How do you decide whether to cover a PR claim or to ignore it? Is a press release or event ever worth covering just to debunk it?
Emily: Coverage is usually pretty easy to assess, based on the importance to your readership, and availability of staff to cover, and competing news of the day. Yes, press releases are worth covering even if just to debunk a claim.
Hal: I disagree, mostly because I think it’s a waste of the reporter’s time. Reader attention is precious, and I’m not sure it’s worth wasting it to tell them something isn’t true. Go out and find something that is. The obvious exception is if the claims are actually dangerously wrong.
Nidhi: We’ve gone back and forth about debunks a fair bit in the BuzzFeed News science [team’s] Slack. If the event or announcement looks like it may be big news no matter what, that’s a reason to cover it skeptically. If an event or announcement is looking like it may get a lot of uncritical coverage, that’s another reason to do the skeptical take. Also: Is it a bad claim that may change people’s behavior? Say, will it influence decisions they make about their kids’ health? That’s a reason to debunk. The conundrum, of course, is this: Does it make sense to give a bad claim that extra oxygen?
Sarah: I don’t write about the vast majority of press releases I receive as stories of their own, but I do use them to build context in my head for the state of a given field or industry, and I archive them so that later, when I do write about the topic, I can easily peruse the latest developments and draw them together. I wait for the archive of press releases to coalesce into a cohesive picture of something larger going on in a field rather than covering those pieces in isolation.
As far as debunking, to be honest, I’m not sure where I stand. The science on how effective debunking is is far from settled. And usually I ask myself, “What new information could I be getting into the world, if I were not spending today proving ‘X is not true’?” Often, both the pro-press-release/event coverage and the debunking cycle fall to staff reporters, rather than freelancers, of which I am one. So if I’m going to even think about covering something that every other science writer on the planet got an email about, I have to think of a different way to write about it—maybe it’s the backstory or historical context, maybe it’s situating it as part of a trend, maybe it’s focusing on the person or a place more than the science. Especially as a freelancer, it’s been a helpful way to go about pitching. It’s harder to get scooped if you found the story yourself and have a (semi)unique take.
Knvul: What other advice do you have for journalists covering areas where there is a lot of cheerleading by the media?
Nidhi: Think about what a source may have to gain from making and publicizing any claim. If it’s any of a) more money b) more success c) more power, treat those claims with appropriate skepticism. Any area with a lot of cheerleading is also a plum opportunity for a reporter to show up, ask some basic questions, and set the record straight.
Emily: It’s really important when covering medical meetings to not just write a story based on a press conference. This can be very misleading and lead to blatantly incorrect headlines. If possible, always go to the actual presentation of data including the question-and-answer session and seek outside comment. Do not rely just on a company and a lead investigator of a medical study to understand results. If your publication requires sensationalizing medical claims, find another position. Also, and this has been talked about a lot lately, do not exaggerate results based on animal studies.
Hal: Stop covering the parts of your beat where the coverage has become uniform and universal. Find a new edge. By the time the press has piled in on a topic, it’s time to find a new story (I feel a little bit like this about Facebook coverage right now). Once you do, you can come back to the topic with fresh eyes. While I understand that journalists are managed and organized on beats, as much as possible I think we should let our interests drive what we write about. Ignore the emails in your inbox, ignore what other people are writing about, as much as possible. Find what you’re interested in and follow those threads.
Sarah: I think it’s more fun not to cheerlead. “Yay, science!” or “Yay, technology!” may produce quick, clickable stories. And most people, especially science journalists, probably got into their beat because it’s a topic they like, and maybe even studied. So it’s understandable that we’d all be excited about whatever we’re covering. But I think it’s important to look critically at our own excitement, as well as at the claims or discoveries.
I, like a lot of science journalists, started out thinking of myself as a “translator,” whose job was to explain. But I don’t see it that way anymore. Explaining complicated topics in a way that a layperson can understand is just one of the tasks I now feel like I need to take on in a given story. I also need to find evidence that the story is, in fact, true. I need to look at funding, bias, influences, methods, applications, and who else is doing, has done, and will do similar work.
As far as advice, I guess I’d say, “Believe nothing (without hard evidence)! Trust no one (until you’ve verified what they’ve said)!” I think that’s the best way to serve readers. Plus, you get to feel like a detective, which I think we all secretly want as journalists.
Knvul Sheikh is a freelance writer and a TON fellow sponsored by the Burroughs Wellcome Fund. Her work has appeared in The Atlantic, Audubon, National Geographic, Popular Science, Scholastic, Scientific American, and more. Knvul has lived in the foothills of the Himalayas in Pakistan, swum in the tropical waters of Singapore, and backpacked across the South Island of New Zealand. She is currently based in New York City and can be found on Twitter @KnvulS.
