Phase 3 trials consider that infections with only mild symptoms qualify as meeting the primary endpoint definition. None of the trials currently under way are designed to detect reduction in serious outcomes such as hospital admissions, use of intensive care or deaths.
Neither do trials address the question as to whether vaccines reduce transmission; while few trials consider whether the vaccine will benefit people who are most vulnerable to the disease: such as the elderly and immunocompromised people.
Peter Doshi, Associate Editor at the BMJ and author of the piece, says: “Rather than studying severe disease, these mega-trials all set a primary endpoint of symptomatic COVID-19 of essentially any severity: a laboratory positive result plus mild symptoms such as cough and fever count as outcome events. These studies seem designed to answer the easiest question in the least amount of time, not the most clinically relevant questions.”
Yet he emphasises that, with the early publication of protocols from major players, there’s an unique opportunity to drive change. “We still have time to advocate for changes to ensure these trials investigate the questions that most need answers,” he says.
Severe illness
One of the reasons that COVID-19 vaccine trials are not assessing severe events is simply a case of numbers. Severe illnesses requiring hospital admission – which only happens in a small fraction of symptomatic cases – would be unlikely to occur in significant numbers in trials.
Even a trial involving tens of thousands would only see relatively few severe cases (figures from April suggest hospitalization in 3.4% of symptomatic cases).
“Hospital admissions and deaths from COVID-19 are simply too uncommon in the population being studied for an effective vaccine to demonstrate statistically significant differences in a trial of 30 000 people," writes Doshi. "The same is true of its ability to save lives or prevent transmission: the trials are not designed to find out.”
In the trials, final efficacy analyses are planned after just 150 to 160 'events' — any positive indication of symptomatic covid-19, regardless of severity of the illness.
A success criterion of 50% efficacy against the primary endpoint (with a confidence interval that includes efficacy as low as 30%) has been agreed by regulators.
The right population groups?
Doshi says there are ‘large gaps’ in recruitment of different types of people in phase III COVID-19 trials.
Only two trials (from Pfizer and AstraZeneca in the UK) are enrolling children less than 18 years old. All exclude immunocompromised people and pregnant or breastfeeding women, and though the trials are enrolling elderly people, “few or perhaps none of the studies would seem to be designed to conclusively answer whether there is a benefit in this population, despite their obvious vulnerability to COVID-19.”
Published protocols
Moderna, Pfizer, AstraZeneca (in the US) and Johnson & Johnson have all published their Phase III study protocols.
“While protocol transparency is not new—and is already common for high impact trials —transparency in real time is an important development and a great win for public trust," says Doshi. "We may not like what we read, but with real time sharing of full protocols comes an unprecedented space for translating critique into action to improve trial design mid-stream.”
‘We have a historic opportunity for the democratisation of science’
Doshi says the early publication of protocols offers a historic opportunity for researchers to collaborate and create the best trials possible.
“The COVID-19 vaccine protocols should be scrutinised by the widest possible readership, to open a critical discussion of many questions about their design and conduct," says Doshi.
"These include why children, immunocompromised people, and pregnant women have been excluded from most trials; whether the right primary endpoint has been chosen; whether safety is being adequately evaluated; and whether gaps in our understanding of the clinical implications of pre-existing T cell responses to SARS-CoV-2 are being addressed.
“We still have time to advocate for changes to ensure these trials investigate the questions that most need answers. While the notion of “moving the goalposts” during ongoing trials may raise concerns, meaningful, scientifically valid, transparently reported amendments to protocols that are made to improve the value of trial results should always be welcomed.
“Critical appraisal of clinical evidence traditionally occurs after the publication of studies, and generally finds much to criticise. Early release of full trial protocols offers a historic opportunity for the democratisation of science. The COVID-19 vaccine trials may not have been designed with our input, but it is not too late to have our say and adjust their course. With stakes this high, we need all eyes on deck.”
Does the flu vaccine lower mortality?
The logic is that if a vaccine is shown to reduce symptomatic disease, it will also protect against serious outcomes.
However, while influenza vaccination has been widespread for decades, no studies have ever looked specifically at whether vaccination lowers mortality. Although randomized trials have shown an effect in reducing the risk of symptomatic influenza, trials have never been conducted in elderly people in the community to see if they save lives.
“Unless we act now, we risk repeating this sorry state of affairs with COVID-19 vaccines,” says Doshi.
Source: BMJ 2020; 371 doi: https://doi.org/10.1136/bmj.m4037 (Published October 21, 2020)
