Researchers: COVID-19 Pandemic Has Disrupted Data Used to Assess Health System Performance

A team of healthcare policy researchers has analyzed one aspect of the COVID-19 pandemic, and determined that, in bulk, the data used to help clinician leaders and health system leaders to assess clinical performance and make decisions, has been significantly disrupted by the pandemic.

“Navigating Through Health Care Data Disrupted by the COVID-19 Pandemic,” appeared on October 12 in JAMA Internal Medicine online. It was authored by Kayoko Shioda, D.V.M., M.P.H., of the Department of Epidemiology of Microbial Diseases at the Yale School of Public Health; Daniel M. Weinberger, Ph.D., of the Center for Outcomes research and Evaluation at Yale-New Haven Hospital; and Makoto Mori, M.D., of the Division of Cardiac Surgery, in the Department of Surgery, at the Yale School of Medicine (all in New Haven, Connecticut).

“The coronavirus disease 2019 (COVID-19) pandemic created abrupt changes in the health care system and health of the general population,” the researchers write. “The disruptions have influenced patient-, hospital-, and physician-level decisions and performance. For instance, the number of patients who sought health care for emergency cardiovascular conditions declined1; outcomes of certain diseases, such as cancer, may have worsened2; and surgical procedures were reserved for patients with emergency conditions. Such changes are inevitably reflected in the health care data for future research. Analytic methods commonly used in clinical trials and observational studies, however, are often based on assumptions that these factors remain stable over time. Therefore, researchers and others interested in research should be cognizant of the unique characteristics of the data generated during the pandemic, and recognize that some studies may not produce reliable results. As some study designs are more likely to be affected, analyses should be appropriately adjusted to allow valid inferences to be made.”

What’s more, the article’s authors write, “The COVID-19 pandemic has influenced multiple levels of health care, including the composition of patient characteristics (e.g., sociodemographic, disease severity, comorbidity), clinician-related factors (e.g., center volume, patient load distribution among physicians, hospitals, and health systems), resource utilization (e.g., intensive care unit admission), and clinical outcomes (e.g., patient-reported outcomes, mortality, readmissions). Such temporary changes matter in ongoing and future research intending to inform patient care, because data generated during the pandemic may be markedly different for these key characteristics. For instance, certain demographic groups, such as women, African Americans, and Hispanics, disproportionately became unemployed and lost employment-based health insurance during the pandemic; the resulting perturbation of payer-based claims data is likely significant and uneven across sociodemographic strata.”

In that context, they write that, “Unlike local disruptions of the data from natural disasters (e.g., earthquakes, hurricanes), economic collapse (e.g., the Greek government debt crisis starting in 2009), or health care worker strikes, the global scope of COVID-19 across diverse groups of people may create unique challenges to study designs that are unable to account for the local disruption by using unaffected age groups, regions, or countries as controls. In addition, the expected use of data sets combining the data generated before, during, and after the pandemic may obscure the populations to which the study findings apply.”

How might healthcare research be impacted right now by all these disruptions that the healthcare system currently undergoing during the pandemic? One clear example, the say, is around clinical predictive models, “such as a model estimating the risk of 30-day postoperative mortality that was calibrated to the pre-pandemic data, [which] may perform poorly in predicting the outcome for operations performed during the pandemic. Such models are unlikely to be calibrated to the increased background mortality rate during the pandemic, patient selection for triaged operations, and the increased risk of mortality associated with preexisting respiratory illness in patients with perioperative severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection. Furthermore, models fitted to data on operations performed during the pandemic might not be accurate in the post-pandemic setting.”

What can be done to adjust for these disruptions? “Potential approaches to data disruptions during the COVID-19 pandemic include the following: (1) excluding the period of disruption; (2) using an analytical approach that can appropriately adjust for the disruption; and (3) relying on carefully designed randomized trials,” the researchers state. “Simply excluding the period of disruptions can be a challenge. Although the COVID-19 case numbers may roughly estimate this period, the duration of latent consequences for data and for the health care system likely exceeds a period defined by case counts. Identifying such a time frame is further complicated by the additional waves of COVID-19 infections in various parts of the world and the potential for the pandemic to become seasonal.6 Considering such a complex interplay, data-driven technique to identify the timing, magnitude, and pattern of changes (e.g., sudden shift in mean, increase or decrease in the rate of change), may become increasingly important in post-pandemic research. For example, change-point analysis detects shifts in the average number of cases or the rate of change and determines when these shifts occur.” All of this will affect data modeling of all types, whether for care management purposes, or for clinical research purposes, they note.

Ultimately, they conclude, “Understanding the magnitude and the period of perturbation is essential for research relying on health care data generated during the COVID-19 pandemic. Although accounting for the pandemic-related changes may be feasible in some instances, the possibility that some study designs may be unable to produce reliable results should be acknowledged. It is important for consumers of such data, including the researchers, reviewers, journal editors, and readers, to recognize the uniqueness of the data generated during the pandemic and evaluate future studies accordingly.”