When should we trust the evidence and the experts who interpret it for us?
A new study makes headlines. It shows a diet works, a supplement sharpens memory, a teaching method transforms reading scores or a management practice improves team productivity. So we adjust how we eat, parent or work. We follow the science.
A few months later, another headline overturns the first. Eggs were bad for us, then good again. Red wine protects your heart, or does it? Dark chocolate, low-fat diets, standing desks, masks during a pandemic: Recommendations based on the science keep changing. Eventually, people stop trusting what “studies show.”
But that is how science works: Evidence accumulates; early results get tested and some hold up, others don’t. The problem is that we report and act on evolving evidence as if it is final.
In a new article, I argue that this tendency to treat preliminary results as if they were conclusive leads to overconfident policy claims and contributes to the declining trust in science. I offer a new framework to help policymakers classify evidence maturity.
The insights can guide all of us to interpret scientific evidence in our daily lives.
Mature evidence vs preliminary results
Claims about what “studies show” are often delivered with confidence, whether they are based on weak early-stage research or confirmed through many years of study across populations and contexts, accumulating into expert guidance and systematic reviews.
While mature evidence has survived years of scrutiny, a surprising preliminary result could easily be overturned tomorrow. Yet, reporters, advocates, marketers and even researchers do not always distinguish between them.
Politicians, educators and public health leaders build policy on a promising early result, call it “evidence-based,” and then watch the policy fail to deliver as the science evolves.
One study is not enough
The framework I have developed — PERLS (Policy Evidence Readiness Levels) — offers policymakers a structured way to classify evidence maturity and assess what “follow the science” actually supports at each level.
It also offers an intuitive rule of thumb: When deciding how much confidence to have in a claim, ask whether it is backed by a single study, many studies or something in between.
When a claim rests on a single study, or even a handful of studies, treat the results as possible, not proven. The finding might hold up under scrutiny. But it might not.
There are many reasons preliminary results fail to hold in the long run. Some fail simply by chance. Run enough studies and a few will find striking results purely by luck, the way a coin can land heads five times in a row without being a two-headed coin. Keep flipping, and the truth eventually reveals itself.
Other results fail for less innocent reasons, such as poor study designs, mistakes or cherry-picked reporting of results.
What happens at scale?
Even results from carefully conducted studies often fail to hold once scaled.
For example, studies may show that shifting to a four-day work week improves productivity among the closely watched, motivated teams that chose to participate in a study and try such practices. But will the results hold if the practice expands across a broader set of employees and organizations?
Other effects only reveal their real impact over the long run, such as a diet that produces early weight loss but is impossible to sustain, or leads to side effects in the longer term.
The case of phonics
Confidence comes not from a single study but from an accumulation of evidence. Do the same results show up again and again across labs and research teams, and in real-world settings with different types of people? We should have more confidence in a finding that survives many studies than a surprising one that first makes headlines.
This is the scientific process, formalized in fields like medicine, where standards of care require evidence to accumulate into meta-analyses, systematic reviews and ultimately guidelines from respected bodies of experts.
When we skip this process, problems emerge. Many educators used a “studies show” argument when replacing phonics with whole-language reading instruction in the 1980s and 1990s.
Decades later, it is clear now that the preliminary evidence on which this shift was based has failed to withstand scrutiny. Schools are now shifting back to embrace phonics and the science of reading, backed by accumulated evidence and systematic reviews.
Previously, educational policymakers failed to question the maturity of the evidence that was used to justify such a major decision.
Act now, remain flexible
We often have to act before evidence is mature. There is a health scare, a market shift or a kid struggling right now. Acting on the evidence we have is not a mistake.
The mistake comes from making commitments as if preliminary insights cannot change.
So, when should we trust the evidence, and the experts explaining it? This is not an all-or-nothing question. We should ask whether the claims are based on a single study or an accumulated body of research established across many studies. We should vary our trust and actions accordingly.
Comments are closed.