Early in the pandemic, as the research community attempted to quantify the effects of pandemic preparedness, they found something peculiar. Many studies on the effectiveness of preparedness on mortality rates found that the countries who had the highest reported preparedness and capabilities also had the highest levels of mortality. Yet, were these initial analyses accounting for the many variables in response, surveillance, and reporting that affect country-wide mortality rates?
Jorge Ledesma is an Epidemiology Doctoral Candidate at Brown University. Together with Dr. Jennifer Nuzzo and partners at the Nuclear Threat Initiative and the Bill and Melinda Gates Foundation, he co-led research that looked further into the available data and attempted to account for these variables. Their work on comparative mortality rates during the COVID-19 pandemic was recently published in BMJ Global Health. This paper is the first to address those research gaps in country age distribution and underreporting. This analysis, unlike others, uses the data from the Global Health Security Index as a resource in order to better understand the effects of preparedness as opposed to a predictive model.
The Pandemic Center sat down with Mr. Ledesma to dive into the results of this paper and what it can tell decision makers about pandemic preparedness.
The following has been edited and condensed for clarity.
Q: What did this analysis find about the effects that preparedness has on mortality rates?
A: This study has two major findings. The first and most notable one confirms the pre-pandemic hypothesis that pandemic preparedness actually works. More specifically, we found that when we accounted for country-level differences in approaches to COVID surveillance, risk factors, severe illness, and income, we observed that countries that were more prepared for pandemics experienced lower mortality rates throughout the COVID-19 pandemic compared to those that were less prepared. In other words, greater pandemic preparedness levels were associated with lower mortality rates.
The second major finding here was that these results were largely consistent when examining different dimensions of preparedness. Scholars have categorized pandemic preparedness according to six different components: infectious disease detection, prevention, response, health systems, international norms, and risk environment. The results were mainly consistent across these dimensions, which indicates that there are a number of different components of pandemic preparedness that countries can invest in to directly modulate their vulnerability to the current pandemic and any possible future public health emergencies.
Q: Why has there been a disconnect between preparedness and measurable positive outcomes before this research?
A: Up to this point there have been several studies showing that pandemic preparedness is actually associated with greater rates of COVID cases. There was thought to be a paradoxical relationship where countries who are more prepared for a pandemic experienced worse outcomes–that is, better prepared countries reported more cases and deaths compared to countries that were thought to be less prepared. These previous studies have sparked a lot of debates about the utility of pandemic preparedness and how we measure pandemic preparedness across very different countries.
However, it's important to note that these previous analyses had several limitations that may have influenced their assessments. They had yet to fully account for country-specific approaches to COVID surveillance. Studies have actually shown that countries like the U.S. have the ability to implement really efficient mass testing sites, while other countries' ability to implement testing is limited by their respective capacities or policies. For example, some countries can only test very targeted groups, such as those with severe illness or travelers. Because of these different surveillance approaches, we may expect that countries with more capacity to perform diagnostic tests, determine cause of death, and publish data, may show higher COVID cases and deaths than countries with less ability to do so. This is particularly problematic because countries who were able to test and publish generally are those with high levels of pandemic preparedness.
The second biggest limitation here is that previous work has not correctly adjusted for risk-factors for severe illness, namely differing age distributions across countries. Without correcting for the age structure of countries, comparisons of mortality across countries may be biased because we know that populations with a larger share of elderly people have increased vulnerability to severe COVID disease and mortality compared to countries with a younger population.
Q: Could you explain what a comparative mortality ratio is and why it is important?
A: The comparative mortality ratio, or CMR for short, is one way to adjust for a limitation in previous research around comparing populations with older populations. The CMR is often used in demography and epidemiology to adjust, or essentially standardize the mortality rates by age to make the rates comparable across countries with very different age distributions. The ideal age-standardization would require very detailed data on COVID-19 mortality by age for every country in the world, which, unfortunately, is not available. However, the great thing about the CMR is that it borrows information on age-specific mortality for just one country, and the only thing that's necessary is the age distribution of comparative countries. The CMR has been really important for us because it provides us the first opportunity to compare mortality rates that are adjusted for the largest risk factor for severe COVID disease, which is age.
Q: Why do you think other papers missed that adjustment of under-reporting and age structure in their assessments?
A: Other papers, to be fair, did outline these limitations in their analyses but the problem was that during these early analyses there wasn’t yet sufficient data on the levels of underreporting and how to account for age structures. To this day only 22 countries report age-specific mortality, which can really limit their analyses. It wasn’t until the end of 2022, or early 2023, when data was starting to emerge on underreporting. We always knew that these limitations existed, but the data just wasn’t quite there yet for other analyses to adjust for them.
Q: What sparked the idea for this paper?
A: The team that wrote this paper has always been skeptical of previous analyses because of the limitations of the data. It's really interesting to look at the previous work because studies have shown that COVID deaths can be up to ten times higher than what’s reported in places like Sub-Saharan Africa. Other papers have shown that global case detection is only around 7%. There is a lot of evidence around potential underreporting, but again this data wasn’t quite there yet for us to do this analysis.
As we were leading up to this paper, we had a lot of discussions around limitations of previous analyses, and we started to refine our ideas so that once the data was available we could then quickly carry out this analysis.
Q: Did you find any results surprising?
A: What was surprising to me is how consistent the results were across the different dimensions of pandemic preparedness. We assessed six different dimensions [from the GHS Index], and within those dimensions, we have a number of different subcategories. In total there were over maybe close to 50 different categories of pandemic preparedness we were looking at and most of them were associated with reduced mortality. This is a particularly important finding because it allows us to see which capacities we can manipulate or improve on to then improve our outcomes during the pandemic.
Q: According to the paper, risk environment is tied to lower mortality. Can you explain why that might be?
A: The risk environment essentially covers factors that give rise to infectious disease outbreaks within countries. What we've found in the analysis was that the risk environment dimension actually had the strongest effect on reduced mortality rates, compared to the other dimensions.
There are several reasons for this. The primary one is that the risk environment looks at how rapidly you are able to mobilize your capacities and put them into place. What we saw during the pandemic is that countries who did really well, like New Zealand, Australia, and Iceland, the main commonality between them is that they had really strong, rapid, and coordinated real-time responses. Because they were able to efficiently mobilize their capacities, we saw that they were able to substantially mitigate deaths during the pandemic. That's probably the biggest factor in why we saw such a strong relationship between the risk environment and lower mortality rates.
Q: How did the U.S. fare in this analysis?
A: Unfortunately, what we found is that, despite ranking the highest in pandemic preparedness according to the GHS index, the mortality rate in the U.S. was much larger than expected. In fact, out of the total 183 countries in our analysis, it ranked 63rd in terms of mortality rates. Despite the U.S. having the most capacities to deal with infectious disease threats, the country did worse than what was expected. There may be several reasons for this.
The primary one has to deal with the risk environment. It's important to note that how preparedness capacities are utilized and how they are put into action really matters here. As I mentioned, countries like Australia, New Zealand, and Iceland, did exceptionally well during the pandemic. They had strong, coordinated, and rapid responses. Another unique thing about these countries is that they had targeted interventions towards vulnerable populations and used consistent communication messages. However, in the U.S. the approach was a bit more disjointed and was hampered by inconsistent messaging and institutional rules that slowed down distribution of testing and diagnostics.
States had very different COVID mitigation methods as well and all these issues harmed compliance with social distancing and vaccination. The key point here is that, though we can have all the capacities in the world, it's really important how we mobilize capacities and put them in place throughout the pandemic.
Q: How do you see this analysis being implemented?
A: Future studies should be mindful of the potential biases in existing global surveillance that may limit our abilities to track disease burdens and compare countries’ performances. Namely, that case and death data are often incomplete in some places more than others, and that cross-country differences in age or population, may limit analysis of how hard countries will be hit. So any objective assessment of preparedness policies on a pandemic’s impact will require overcoming these limitations. However, a great thing about this potential exercise is that it provides an opportunity to reflect where there are gaps in surveillance data and where there are gaps in pandemic preparedness policies. This provides an excellent opportunity to improve future analyses and start to invest in pandemic preparedness response, capacity, and policies.
Q: Has this paper sparked any future research ideas?
A: I'm planning on doing my whole dissertation around pandemic preparedness because of this analysis. I have a series of studies that I’m working on currently. The first will be around whether pandemic preparedness policies can help us improve disease detection, for both infections and deaths. That paper is actually currently going through peer-reviews and hopefully will come out pretty soon. The major finding from that study is that, not surprisingly, having a greater number of pandemic preparedness policies allows one to better detect and enumerate COVID infections and deaths across the world.
The second study will be around the question: can these pandemic preparedness policies help us improve indirect consequences of the pandemic? As we saw during the pandemic, there were increases in mental health concerns, disruptions in healthcare services, and increases in financial insecurity and income security. This second study examines whether these policies can help us mitigate or modulate these adverse consequences throughout the pandemic.
Then, the final study will mostly focus on healthcare disruption and ask: can these pandemic preparedness policies help us mitigate disruptions in health services that we saw during the pandemic? Emerging evidence has shown that people were missing healthcare appointments and surgical procedures, and any delays in these services can increase morbidity and mortality. This final study is looking at whether these policies can help us mitigate these disruptions.