How can anyone decide on the best course of action in a world full of unknowns?
There are few better examples of this challenge than the COVID-19 pandemic, when officials fervently compared potential outcomes as they weighed options like whether to implement lockdowns or require masks in schools. The main tools they used to compare these futures were epidemic models.
But often, models included numerous unstated assumptions and considered only one scenario – for instance, that lockdowns would continue. Chosen scenarios were rarely consistent across models. All this variability made it difficult to compare models, because it’s unclear whether the differences between them were due to different starting assumptions or scientific disagreement.
In response, we came together with colleagues to found the U.S. COVID-19 Scenario Modeling Hub in December 2020. We provide real-time, long-term projections in the U.S. for use by federal agencies such as the Centers for Disease Control and Prevention, local health authorities and the public. We work directly with public health officials to identify which possible futures, or scenarios, would be most helpful to consider as they set policy, and we convene multiple independent modeling teams to make projections of public health outcomes for each scenario. Crucially, having multiple teams address the same question allows us to better envision what could possibly happen in the future.
Since its inception, the Scenario Modeling Hub has generated 17 rounds of projections of COVID-19 cases, hospitalizations and deaths in the U.S. across varying stages of the pandemic. In a recent study published in the journal Nature Communications, we looked back at all these projections and evaluated how well they matched the reality that unfolded. This work provided insights about when and what kinds of model projections are most trustworthy – and most importantly supported our strategy of combining multiple models into one ensemble.
Collecting projections from multiple independent models provides a fuller picture of possible futures − as in this graph of potential hospitalizations − and allows researchers to generate an ensemble.
COVID-19 Scenario Modeling Hub, CC BY-ND
Multiple models are better than just one
A founding principle of our Scenario Modeling Hub is that multiple models are more reliable than one.
From tomorrow’s temperature on your weather app to predictions of interest rates in the next few months, you likely use the combined results of multiple models all the time. Especially in times like the COVID-19 pandemic when uncertainty abounds, combining projections from multiple models into an ensemble provides a fuller picture of what could happen in the future. Ensembles have become ubiquitous in many fields, primarily because they work.
Our analysis of this approach with COVID-19 models resoundingly showed the strong performance of the Scenario…
