It is very exciting to hear another positive story about vaccine trial results – a good vaccine is the most likely way of ending the pandemic.
Last week, interim results from Pfizer suggested its vaccine reduces cases of COVID-19 with 90 percent efficacy. Now Moderna has gone one better, with interim results showing nearly 95 percent efficacy for its vaccine – with hints that it may protect against severe disease.
Neither have reported any serious safety concerns and have tested their vaccines in tens of thousands of participants.
With so many COVID-19 vaccines in development, more results are likely to follow in the coming months. Their headline figures may, like these, be very impressive, but it’s necessary to dig deeper to find out exactly what any new results mean.
With that in mind, here are six questions to ask about any new vaccine trial result.
1. Does this mean the vaccine is safe?
Almost certainly yes if it has successfully passed through a phase 3 trial with thousands of participants. Vaccines do not get this far if there are any major doubts about safety.
Historically, pharmaceutical companies have been able to suppress negative results, but it’s now legally required for all trials to post their results so that other scientists can review them.
As a consequence the sector is generally far more trusted than it used to be, although we should still be cautious if only interim results are being reported.
Some people are concerned that COVID-19 vaccines have been produced with unprecedented speed; however, the vast majority are based on platform technologies with excellent safety profiles.
There are a few newer technologies being used, but the clinical trial and regulatory process is extremely rigorous and will pick up the majority of potential complications fairly early on in development.
Of course, it is still difficult to know yet about long-term side-effects, but these are rare for vaccines, and any risk is normally significantly lower than the risks from getting the disease being vaccinated against.
2. Do the headline figures reflect what the trial was designed to measure?
Trials often measure many things, but there is always a single primary research question or objective that a trial has been designed to answer.
Trials will also have several secondary research questions, but answering these is not considered a mark of success. If you test enough different objectives, a few will always be met due to blind chance.
Misrepresenting trial data in this way is a form of research misconduct called p-hacking. You can find out the primary and secondary objectives of any trial by checking a clinical trial registry.
Again, it is important to consider whether these are interim results. Although such results can be promising – as Pfizer and Moderna have shown – they are not guaranteed to be the final result.
3. Did the trial measure the right thing?
Determining what counts as a medicine or drug “working” can be quite complicated for many diseases.
But for vaccines, the question to ask is quite simple: did people who had the active vaccine get the disease?
Any measure that’s more complicated than this (often referred to as a surrogate outcome) should be treated with caution.
4. Who was the vaccine tested on?
Are the results of a trial transferable to the real world? Here it’s important to understand the difference between a population (in this case everyone who can catch COVID-19) and the sample of that population who took part in the trial.
In many cases, trials use two carefully matched (and so comparable) samples in carefully controlled conditions.
One is given the vaccine and the other a placebo (such as saline injection or an already developed vaccine for another disease) to control for the effect of participants thinking they have been vaccinated – which does have an effect.
In phase 1 trials, safety concerns mean that samples are generally made up of young and fit people with few health concerns, who are probably not representative of the overall population.
However, as trials progress into later phases and get bigger, researchers try to ensure a more representative sample of the population.
This is why the final-stage (phase 3) trials are so important, as the sample is chosen to represent the population that the vaccine is targeted at.
Formal publications of trial results normally provide a table describing who was in the sample, and often efficacy rates for the different groups (broken down by sex, age and so on).
Unfortunately, the headline efficacy figure (95 percent for instance) may not apply evenly across the population.
This is very important for COVID-19, as we know older people are much more vulnerable. We therefore shouldn’t read too much into any results until we can see an age breakdown for efficacy.
5. Will the vaccine be usable?
Before we get too excited, some practical questions must be asked. How much will the vaccine cost? Can it be made in bulk? Is it easy to transport and store? And how many boosters will be needed?
These logistic problems (for instance, the requirement to be stored and transported at very low temperatures) can easily prevent a new vaccine getting into the clinic.
6. Can we trust what’s being reported?
It’s an increasingly important skill to identify between reliable and unreliable sources. Social media is often superficial and prone to spreading misinformation.
On the other hand, journal articles and clinical trial registries can be hard to interpret for anyone except specialists.
Trusted journalism is the answer. Seek out publications with editorial oversight and a track record of reliable scientific and medical reporting. Reading more than one interpretation can help you get a balanced view.
It’s also important to ask where a journalist found the information they are reporting on.
Referencing results published in peer-reviewed journals is a good sign – it shows some rigorous fact checking has occurred.
Be careful if an article’s main sources seem to be preprints (papers not yet peer reviewed) or other so-called grey literature, such as press releases or company reports.
Likewise, be careful if the main source seems to be interviews or quotes from people with PhDs or impressive sounding job titles.
A quote from a scientist in an interview is not equivalent to a quote from the same scientist in a peer-reviewed academic paper.
Simon Kolstoe, Senior Lecturer in Evidence-Based Healthcare and University Ethics Advisor, University of Portsmouth.
This article is republished from The Conversation under a Creative Commons license. Read the original article.
