Over the past year, one message has been made clear: trust in science. Evidence, and only high quality evidence, will form the basis of policy. How did this influence the coronavirus vaccination campaign? On the one hand, there has been a strict respect for scientific rigor when it corresponds to the desired narrative. On the other hand, scientists can differ in the way they interpret science. For example, some stressful treatments are only “evidence-based” when they have been evaluated in a randomized, double-blind clinical trial and applied using protocols that accurately mirror the research studies. Others are willing to accept evidence from modeling exercises based on questionable assumptions. Yet when advocating for greater acceptance of vaccines, the scientific standard is far from uniform. When communicating with the public, the same scientists may apply different standards depending on whether the study findings match the desired message.
Why would the world’s greatest scientific thinkers apply the label of “good science” so inconsistently? Perhaps the best explanation is what psychologists call confirmation bias, which is the tendency to interpret observations or data in a manner consistent with previously established beliefs and values. Thousands of studies over the past 50 years show how confirmation bias obscures findings in science, the arts, politics, court decisions, finance, and medicine. Nobel laureate Daniel Kahneman, PhD, has contributed to many examples showing how, even among the most experienced scientists, preconceptions color the interpretation of data and events. And worse yet, previous beliefs influence the way we look at information. Studies have consistently shown that people uncritically accept evidence that confirms their beliefs, while subjecting unconfirmed information to rigorous skeptical evaluation.
Before continuing, we want to be clear: we support the widespread deployment of coronavirus vaccines. We were both vaccinated as soon as we were eligible. But we are concerned that biases may cloud the interpretation of evidence where scientific uncertainty persists.
Let us examine how the trend of confirmation bias played in the evaluation of studies supporting vaccines. For example, days after a suggestion of increased myocarditis following vaccination in young Israeli men, CDC director Rochelle Walensky, MD, MPH ignored the evidence from the CDC’s own surveillance systems and jumped on the ball. on the conclusion that vaccines posed no threat. But now the FDA has added a warning about the risk of myocarditis after vaccination with mRNA injections, and the CDC has agreed to update its fact sheet.
NIH Director Francis Collins, MD, PhD, who by anyone’s standards is a model of personal and scientific integrity, issued a Blog with the headline, “Studies Confirm COVID-19 mRNA Vaccines are Safe and Effective for Pregnant Women.” The evidence was based on two studies. One of the studies included only 30 pregnant women and did not measure maternal or child health outcomes. The small sample size is a problem. Imagine concluding that maternal age is unrelated to Down’s syndrome based on 30 women aged 35-40. Down syndrome, which occurs in about eight out of every 1,000 live births in 40-year-old mothers, would most likely be overlooked. the second study included only 84 vaccinated women who had given birth. Examinations showed that the placentas of these births were comparable to those of a group of women who had not been vaccinated. These two studies, comprising a total of 114 pregnancies, were then generalized to all women and to birth outcomes rather than surrogate measures of immunity or placental pathology.
The evidence used to reassure men may be even weaker. In June, JAMA published a study which was designed to determine whether mRNA vaccines decrease fertility. The survey involved a total of 45 young volunteers (median age 28). The semen was collected before and after the vaccination. There was a modest increase in sperm concentration, motility and volume after the vaccine. No data on pregnancies, live births or neonatal complications were available. MedPage today reported the results under the headline “Hopeful Dads Can Relax About COVID Vax: No Link to Infertility.” A quote from the lead author narrowed the methodological limitations: â… even though the number 45 is small, we are confident that we can generalize this to the rest of the population. He went further to express his confidence that the Johnson & Johnson and Novavax vaccines, which were not evaluated in the study, would also not affect sperm count. Urology schedules reported, âStudy Shows COVID-19 Vaccines Do Not Affect Male Fertilityâ without raising a single question about methodological limitations. CNN, under the title “The sperm count is not damaged by the Covid-19 vaccine, according to the study,” quoted several experts who reassured the men that the study dispels any concerns about the effects of the vaccine on fertility. Yet these small studies exert a disproportionate influence because JAMA publications often receive a lot of media attention.
Now let’s do a thought experiment. Suppose the study shows a decrease in sperm concentration or motility after the vaccine. Would have JAMA accepted the paper? Or would the reviewers say: 1) there were only 45 subjects, 2) he was using a convenience sample that is not representative of the US male population, 3) there is no had no control group, 4) results were surrogate markers, not actual measures of reproductive success, and 5) follow-up was limited to 70 days after the second dose.
The list goes on. The concern, of course, is that confirmation bias is at work. JAMA adheres to very high methodological standards for articles that challenge the dominant narrative. But for studies that reinforce mainstream wisdom … not so much. To be fair, we don’t know of any evidence that vaccines negatively affect fertility. But we need more time and more evidence to say that vaccines have no effect on birth outcomes. This is why Pfizer, Moderna, Johnson & Johnson and CDC have remained cautious – CDC says pregnant women can be vaccinated and should discuss any questions with a health care professional; NIH also fair launched a study for more information on the vaccine for pregnant women.
This brings us back to the complicated quagmire of vaccine defense, science, and confirmation bias. There is no doubt that leading doctors and scientists, eager to encourage vaccination, sincerely believe that vaccines do not affect fertility or the outcome of birth. Does the obvious justify the enthusiasm? Vaccine makers have made it clear that there is not enough data to determine whether vaccines are safe and effective for pregnant women. They are only now starting studies that will provide sufficient evidence on safety. Politics must be based on sound science. We must recognize our fallibility in interpreting evidence: When we use science to warn or reassure the public, even the best scientists must recognize that we are all potential victims of confirmation bias.
Robert M. Kaplan, PhD, is a faculty member at the Center for Clinical Excellence at Stanford University and Emeritus Research Professor at the Fielding School of Public Health at UCLA. Rose McDermott, PhD, is the David and Marianna Fisher University Professor of International Relations at Brown University.