In 2018, science writer Daisy Yuhas became the features editor for Sapiens, which specializes in covering research in the field of anthropology. She quickly became fascinated: The work was a world away from the kind of papers she had previously covered for publications such as Scientific American or Audubon. Like many science journalists, Yuhas had learned how to read, analyze, and understand quantitative research—the kind of number-heavy research, often derived from controlled experiments, that dominates scientific fields such as astronomy, microbiology, and geology.
But with some exceptions (such as archaeology and the study of humanity’s biological ancestry), the anthropological research she was covering now was different. Instead of conducting experiments, researchers spent a long time observing communities’ behaviors and interactions. Sometimes, entire papers were based on a handful of lengthy conversations with key players in the community the scientists were studying. This qualitative approach was new to Yuhas, but it extends well beyond anthropology. It underpins some subdisciplines of fields such as sociology, political science, history, and economics, where data might come from researchers’ interpretations of old documents; and of psychology, the health sciences, and linguistics, in which subjects’ perceptions of the world might take center stage.
As an editor for Sapiens, Yuhas noticed that many of the writers she worked with found it tricky to cover qualitative research, which tries to decipher reality by understanding its non-numerical aspects, such as how social organizations are created and maintained, how those structures influence people’s opinions, emotions, and behaviors, or how concepts such as “family” can vary widely between two cultures. “I have so many people who pitch me, and they say, ‘I’ve never really looked at this kind of research before,’ and they feel like it’s so different,” Yuhas says.
Science journalists are rarely taught how to evaluate the credibility of qualitative papers. But such research can be fascinating and worthy of coverage—and can add depth and complexity to many science stories, Yuhas says. In the case of cultural anthropology, researchers might look at why people behave a certain way in certain contexts. Why, for example, do some people choose to avoid the COVID-19 vaccine, despite the scientific evidence supporting its safety and efficacy? Or, what meanings do people attach to the word immigrants, causing them to perpetuate negative stereotypes? In this kind of research, anthropologists ask questions to reset the way we look at these situations, Yuhas explains. “And by raising those questions, they also raise different solutions.”
Quantitative and Qualitative Research: Sisters, but Not Twins
Despite how distant from one another quantitative and qualitative research sometimes seem, at a fundamental level they are based on the same ideas, says Rik Peels, a philosopher and theologist at Vrije Universiteit Amsterdam (VU Amsterdam) who studies issues of research integrity. They’re both empirical, meaning that they’re based on carefully observing the world, whether by using a CT scan, a telescope, or in-depth interviews with subjects. In all cases, scientists gather data and then use it to draw conclusions.
Whether the research methods are quantitative or qualitative, a good paper will guide the reader through every logical step of its argument.
The similarities also go beyond these core principles, says Roberto Erick Arceo López, assistant editor of the Revista Mexicana de Sociología (Mexican Magazine of Sociology), which is based at the Universidad Nacional Autónoma de México (UNAM). Arceo López says that whether the research methods are quantitative or qualitative, a good paper will guide the reader through every logical step of its argument: It will present the question or hypothesis guiding the research, explain how other people have explored the question, describe the steps the authors took to solve the question, share the data they gathered and which methods they used, and finally, offer the authors’ conclusions, in which they explain if the data answered the question they initially posed.
Another aspect both types of research have in common is that, to be robust, they have to be systematic, explains Tamarinde Haven, a researcher on integrity in the humanities at VU Amsterdam. This means that, as with any scientific endeavor, qualitative research results should arrive after a thoughtful, organized process of gathering and analyzing information.
However, science journalists covering qualitative studies do need to adjust the criteria they use to evaluate them. Many of the important parameters that signal the quality of a quantitative paper—like sample size or statistical soundness—are either absent or not as relevant when evaluating qualitative research. After all, these studies ask very different questions than quantitative studies do, and so their way of answering those questions is also quite different.
Dethrone Numbers
If science journalists ignore that qualitative research concerns different aspects of human behavior from those studied with a quantitative lens, they can end up dismissing important research that, with time, could become a pillar of our understanding of society. For example, in 1984, a Black social scientist named Philomena Essed published a short book called Everyday Racism. In it, she presents the results of a series of in-depth interviews with 55 Black women living in the Netherlands and California.
Essed argued that all of the women experienced “recurrent, systematic, and familiar practices” of racism—not outright racist attacks, but “normal” interactions that were embedded in the culture, she argued. Journalists and some researchers heavily criticized her work. The reason: Essed, they claimed, didn’t present enough interviews to say there was such a thing as “everyday racism.”
Essed’s case is a perfect example of journalists and researchers imposing criteria for evaluating quantitative methods onto qualitative research, Haven says. “I think the first thing that [science journalists] need to do is to be aware that not everything is expressed in numbers.” Sample size (how many people, for example, were involved in a medical study) is a valid parameter to evaluate how credible quantitative research is. After all, generally speaking, these types of studies try to reach universal—or at least generalizable—facts about a phenomenon or group of people.
Generally, most qualitative research does not aim for universality; instead, it tries to understand specific groups of people at a determined point in time and place.
But generally, most qualitative research does not aim for universality; instead, it tries to understand specific groups of people at a determined point in time and place. The intention is not usually to generalize, says Haven, but “to point us towards issues that have been overlooked, or that cannot even be investigated with quantitative methodology.”
In Essed’s case, her intention wasn’t to claim that “everyday racism” was something experienced by all or most Black women (though this is something that statistical analyses proved years later) but to show its existence and to examine the ways in which it worked. Qualitative research can be a starting place to reveal an issue that, further down the road, might be better understood through quantitative data.
Instead of focusing on sample sizes, Haven says journalists should check whether the researchers reached their conclusions based on the data collected, whether that was during interviews, group sessions, or direct observations. She pays special attention to quotes from interviews that researchers include in their papers, and asks herself whether the quotes seem to convey the meaning that the authors have inferred.
Another telltale sign of high-quality research is when researchers allow study participants to see the transcripts or summaries of their interviews before publication. This research practice, called “member checking” and often included in a paper’s methods section, gives participants an opportunity to provide feedback on researchers’ interpretations of their conversations, and modify anything that might have been misunderstood.
In qualitative research, oftentimes the methodology has to be adjusted as the research advances (for example, if a particular question in a questionnaire is not working, researchers might want to change it). Oftentimes, these changes are recorded in memos (sometimes called “side notes”), where researchers write their impressions and expectations throughout the process. Because of this, memos also help evaluators and researchers not involved in the study to retrace the reasons behind any decision taken during the course of the investigation.
Sometimes, researchers don’t rely on interviews, but rather on active, long-term observation, taking notes about the interactions and behaviors of people in the community they’re studying. In those cases, memos can help the researcher to stay open during every interaction with the group, instead of getting subconsciously fixated on one particular answer. “It can be very easy at the start [of the research] to think, Oh, this is it. This is just going to be the answer to my question,” Haven explains. Making those thoughts explicit in their memos allows researchers to check their assumptions and biases during fieldwork.
These side notes, also markers of good fieldwork, often contain sensitive information about participants, so they tend to be excluded from papers. But a note regarding the existence of memos sometimes appears in a study’s methods section, study preregistration, or protocol. If not, science journalists can ask researchers about them during interviews.
The View from Somewhere
One of the key ways in which qualitative research differs from quantitative research is the researcher’s place in the study. In most quantitative science, you could, in principle, replace the scientist with someone else; if they apply the same methods, the conclusions should be the same. This is, as VU Amsterdam’s Peels points out, the principle behind replication—the process of checking the robustness of scientific conclusions by repeating the experiment.
In most qualitative frameworks, in contrast, researchers acknowledge that they are not objective observers, but simply humans trying to gather data on other humans, and that their own biases and lived experiences could influence their results. Robust studies will include an acknowledgment of these biases, often as a section of its own or as part of the methods section. This section will state, very clearly, the theoretical framework and concepts under which the researcher will analyze the data. “In my field, philosophy, you might work with, say, a Marxist paradigm or an existentialist paradigm, or a feminist epistemology—there are various options,” says Peels. “And such background assumptions and paradigms could steer the research, or color certain concepts, or rule out certain questions, or inspire other questions. So that needs to be very explicit.”
Science journalists starting to dip their toes into humanities and social sciences research should take extra time to engage deeply with the internal debates of a field and the competing frameworks within it.
The conceptual frameworks that underlie research in fields such as philosophy, social work, and sociology are the building blocks for qualitative papers, and using them accurately is a primary indicator of the quality of a paper, says Arceo López, from Revista Mexicana de Sociología. “If you bring me research that says that it’s based on a decolonial framework [a framework developed by Latin American scholars trying to de-center European and American points of view of history, economy, gender, or politics], but you only cite European authors, then that’s a red flag,” he says.
And precisely because such concepts are so important, science journalists starting to dip their toes into humanities and social sciences research should take extra time to engage deeply with the internal debates of a field and the competing frameworks within it, says Yuhas, from Sapiens. “If this isn’t your area, over-report a little bit, in terms of learning who the key players are in that field,” she says—just like you would if, for example, “you’ve done a lot of organic chemistry and now you want to start covering ecology and the behavior of capuchin monkeys.”
Getting acquainted with the debates within fields that are new to them could help journalists recognize the language scholars in those fields use, changes in how they do things, and the assumptions that are baked into the research that may not be evident to nonexperts—all details that could be stories in themselves.
Build New Structures
Once you’ve decided if a paper is robust under the criteria of nonexperimental research, you’ve arrived at the most challenging part of covering it: Figuring out how to write about it. “You’re probably not going to be able to tell those really tidy [narratives that say], ‘We thought this, we tested this, we saw this’” that [are] present in the skeleton of most science stories, Yuhas says. “Instead, you’re often telling a story about someone else’s arguments or perspective. [The story] is more like, ‘People generally interpret X in this way, but [this scholar has] a new way of interpreting these observations.’”
A new, fresh way of looking at a problem can be the central thread of a story based in social sciences knowledge.
A new, fresh way of looking at a problem can be the central thread of a story based in social sciences knowledge. A story Sapiens published about the research of University of North Carolina at Chapel Hill medical anthropologist Aunchalee Palmquist, for example, complicates the common medical narrative around mothers who share breastmilk with each other’s babies, a practice that most doctors consider dangerous due to the risk of spreading pathogens. Palmquist gathered the real-life stories of mothers, arguing that the informal exchanges happen as a consequence of the medical establishment’s endorsement of breastfeeding over formula, which has motivated some mothers to avoid formula at all costs. Her research encourages medical practitioners to approach conversations with these mothers with less judgment, helping them to feed their babies safely through shared milk circles.
One of the most difficult aspects of covering qualitative research is probably the task of creating a solid nut graf. After all, in disciplines such as cultural anthropology, linguistics, or history, there’s often no concrete, quantified result to serve as the center of gravity to the story. Instead, Arceo López recommends asking the researcher—or yourself—what kind of relevant information about our society the research provides, or how the new framework used in that particular paper gives information that can help a community deal with a specific issue.
Yuhas says the research’s usefulness is key in the nut graf of many Sapiens stories. The magazine recently ran a story about an anthropologist studying an invasive population of macaques in Florida—and how shifting paradigms in the understanding of how humans relate to other species can help us figure out a way to reach agreements in invasive-species debates.
After more than three years working as an editor in Sapiens, Yuhas says she feels “endlessly excited” about the kinds of questions that working with qualitative research has forced her to ask herself. “You see a lot more variation baked in [to research in the social sciences] than we sometimes, as science journalists, remember is there,” she says. “It’s a reminder that all of us are still part of the exciting, human experiment of gathering information.”
María Paula Rubiano A. is a TON early-career fellow sponsored by the Burroughs Wellcome Fund and a freelance science journalist writing about biodiversity, environmental justice, food, and sustainability for Science, Yale Environment 360, Audubon, Atlas Obscura, El Espectador, and more. Follow her on Twitter at @Pau_Erre.
