Making projections is a difficult game, even in the best of times.
It’s interesting to look back at what we thought about the coronavirus in March, April and May, comparing it to what we know now. Even after the virus had ravaged other countries for a month or two, we still didn’t know what to expect in the United States, and even less about how the virus would impact a state like Utah.
With that caveat in mind, I was curious to check how well the projections from that period held up, both the ones from statistical models and ones created by experts. And while we’re at it, let’s take a look at what experts are projecting for the future.
Projection models like IHME
Remember them? The Institute for Health Metrics and Evaluation at the University of Washington was really the first group to release a coronavirus projection for each state, and it showed both the upsides and downsides of doing so.
The good news was that these researchers received a ton of attention in the media and from the government — the White House was said to be using the IHME’s model to make decisions. The bad news was that being the first with a public model on a novel coronavirus meant the model wasn’t very good.
In particular, the IHME based its projections on the course of the virus in Wuhan, China, which saw a quick and immediate spread followed by a rapid decline after a dramatic and complete shutdown. Expecting the virus to take that shape in the United States wasn’t particularly wise: American shutdowns were always going to be less severe than Chinese ones.
The model also saw huge swings as researchers tinkered with the math behind the model and saw new data coming in. For example, on the first day the model was released, March 25, it projected there would be 619 deaths in Utah by August. Ten days later, it had changed the projection to 185 deaths. Honestly, these kinds of wild swings aren’t unusual in a model which projects a disease that can experience exponential spread, but it led to whiplash all the same.
As of Wednesday, Utah has suffered 327 deaths due to the COVID-19 pandemic.
Nationally, IHME’s first guess was actually its most accurate in terms of number of deaths. On March 25 the model thought the United States would see 162,000 deaths by August 4th. On August 4, we had 156,000. That’s pretty impressive, but the projection had gotten the course of the disease, in Utah and the nation as a whole, all wrong.
Whoops. The IHME predicted a bigger spike in the spring, followed by a massive reduction going into the summer months, and near-eradication by August. That is the path that we’ve seen in some European countries, but that certainly didn’t happen in the U.S. And later models pegged the number of deaths far too low, at 60,000.
A couple of other models are worth mentioning: the Los Alamos National Lab started releasing projections, typically predicting the virus to continue on its current course then fade after six weeks or so. It continually under-predicted the death toll.
The COVIDActNow.org projection was always intended as a scare model, and even its best-case scenarios significantly overshot the impact of the virus. For example, in Utah, it projected 1,000 deaths over three months even if a shelter-in-place order was added. It only was in a few Utah counties, and we haven’t had nearly that many deaths.
Panels of experts
Infectious disease researchers polled by the University of Massachusetts-Amherst were asked to project how the pandemic would play out in weekly surveys delivered from March to June. On March 25, the group of 20 experts projected 245,000 deaths in the United States by the end of 2020, a number that is certainly feasible as we stand. They projected that hospitalizations would peak in April or May, and they actually did in April, with a second smaller peak in July.
Hospitalizations as tracked by the CDC.
I was also impressed with the researchers’ estimates a month later, on April 27. Then, with the pandemic in full swing, they were asked when weekly deaths in the U.S. would go below 5,000. The largest group chose late June among the available options, with the second largest group selecting mid-June. Weekly deaths went below 5,000 for the first time in the week of June 20th, before bouncing back up by mid-July.
Finally, a second panel of researchers were asked about the topic of vaccines in June. In particular, they were asked how many vaccine candidates would be in human trials by August 1. The median number guessed was 17. In fact, there were 26 candidates in human trials on August 1.
Looking forward
The immunization panel was relatively pessimistic about when such a vaccine might arrive. The median prediction for the month at which 100,000 people in the U.S. would be injected with an FDA-approved drug was November 2021. Remember that they didn’t have the results of some of the Phase I/II testing of the leading vaccine candidates, though, which has been relatively promising.
Back before the the June/July spike in cases, the UMass panel of experts was also asked whether or not they predicted a second wave would occur in the fall and 73% said that it would, though I’d be curious to see how they would respond today.
To their credit, IHME continues to reevaluate and reshape its model, putting much more emphasis on the current trajectory of the disease in each location rather than what happened in Wuhan. They also now provide multiple projections depending on different inputs, such as whether or not universal masking is adopted.
And IHME’s new and improved model projects 230,822 COVID-19 deaths in the United States by November 1, and 533 in Utah. That’s relying on three assumptions:
- Relative and gradual easing of social distancing mandates.
- That easing would continue unless Utah faces more than 24 deaths per day, in which the model assumes mandates would restart.
- Masking stays where it’s at: many people wear masks but below universal levels.
If, on the other hand, Utah adopts universal masking — defined as 95% masking in public in every location — then the IHME predicts only 413 deaths in Utah and 198,000 in the U.S.
And that’s the thing: both the experts and modelers are still working with imperfect information, having to make assumptions about how this will play out. We know much more about the disease and how it spreads than we did in March, but we don’t really know how school openings and especially the fall season will impact things — we just haven’t gotten that information yet.
In other words, in six months, we’ll have to compare these projections with reality again. So goes the life of a forecaster.
Andy Larsen is a Salt Lake Tribune sports reporter who covers the Utah Jazz. During this crisis, he has been assigned to dig into the numbers surrounding the coronavirus. You can reach Andy at alarsen@sltrib.com or on Twitter at @andyblarsen.
Follow @andyblarsen