External Source Analysis
ConsumerHealth NZ — Independent health information for New Zealanders consumerhealthnz.org.nz
Independent. Accessible. Yours.
Understanding Research

So a Study Says It Works — But Does It Really?

Every week, headlines tell us some new food, supplement, or treatment "works." But not all research is created equal. Here's how to tell the difference between solid evidence and a headline that got carried away.

[Illustration: scientist looking at data]
A single study — even a well-designed one — rarely tells the whole story. (Illustrative image)

You've seen the headlines. "Coffee linked to longer life." "New drug cuts cancer risk by 40%." "Scientists discover vitamin X prevents memory loss." They're everywhere, and they're genuinely hard to evaluate if you haven't been trained to read research papers.

The good news: you don't need a science degree to be a smarter reader of health research. You just need to know a few key questions to ask.

What Kind of Study Was It?

Not all studies carry the same weight. The type of study matters enormously for how much confidence we should have in the results. Studies designed to find associations between things (observational studies) can tell us that two things seem to go together — but they often can't tell us whether one thing is actually causing the other.

The gold standard for testing whether a treatment actually works is the randomised controlled trial (RCT). In an RCT, participants are randomly assigned to either receive the treatment or not. This random assignment is crucial — it's what allows researchers to claim that any difference between the groups is likely due to the treatment, rather than something else.

Study Types: A Quick Guide

Randomised controlled trial (RCT): Participants are randomly assigned to treatment or control group. Considered the strongest evidence for whether a treatment works.

Observational study: Researchers observe what happens in groups of people, without intervening. Can find associations, but association is not the same as causation.

Case study / anecdote: A report on one individual or a small group. Useful for generating ideas but cannot draw general conclusions.

Systematic review / meta-analysis: Combines the results of many studies. Generally the most reliable overview of evidence.

The Size Problem

Here's something that surprises many people: a lot of published research involves surprisingly small numbers of participants. Studies show that small studies are much more likely to produce results that are simply due to chance — and those chance results are also more likely to be dramatic and therefore newsworthy.

A study of 30 people who took a new supplement and felt better is interesting. It is not, however, evidence that the supplement works for the general population. You'd need a much larger, carefully controlled trial to reach that conclusion.

Watch Out For

Headlines that say "a new study finds..." without mentioning how large the study was, whether it was randomised, or whether the results have been independently replicated. These omissions are often more revealing than what is included.

What Is a Control Group and Why Does It Matter?

A control group is the comparison group — the people who don't receive the treatment being tested. Without a control group, there's no way to know whether any improvement in the treatment group was actually caused by the treatment.

This matters more than you might think, because of something called the placebo effect. Studies consistently show that people who receive a treatment — even an inert one — often report feeling better. The act of being treated, being cared for, and expecting to improve can produce real physiological changes. A good study accounts for this by giving the control group a placebo and — crucially — not telling anyone who received what until the results are analysed.

Has It Been Replicated?

In science, a single study — even a well-designed one — is rarely considered conclusive. Results need to be replicated by independent researchers before they become established knowledge. This is a feature of science, not a bug: it's how the system weeds out fluke results and fraudulent research.

When a finding has been replicated across multiple independent studies, and a systematic review has confirmed the pattern, that's when we can start speaking with more confidence. That's very different from "one study showed..."

Questions to Ask When You See a Health Headline

Was it an RCT, or just an observational study?
Observational studies can suggest associations; only well-designed RCTs can reliably show that a treatment causes an effect.
How many people were in the study?
Larger studies are generally more reliable. Studies of under 100 people should be treated as preliminary.
Has it been replicated?
One study is not enough. Look for whether other researchers have found the same result.
Who paid for it?
Studies funded by companies with a commercial interest in the outcome should be read with extra attention to methodology.

When Headlines Get It Wrong

Science journalism is genuinely difficult. Journalists work to tight deadlines, research papers are often paywalled, and nuance doesn't make great headlines. Studies show that many health stories in mainstream media overstate the strength of evidence, conflate association with causation, and omit important caveats.

This doesn't mean you should dismiss all health news — it means you should read it critically. Look for whether the journalist has spoken to independent experts (not just the paper's authors), whether they've explained the study design, and whether they've noted limitations.

The Bottom Line

No single study settles a question in medicine. Evidence accumulates over time, through multiple well-designed studies, peer review, and systematic evaluation. The body of evidence is what matters — not any individual headline.

Phrases like "studies suggest" or "research indicates" are genuinely different from "it is established that." Knowing the difference makes you a much better reader of health information.