New option could also aid in resource utilization.
Background: SARS-CoV-2 infection has resulted in a high mortality rate, with the majority of deaths resulting from respiratory failure. As waves of the pandemic continue to overwhelm healthcare systems throughout the world, a pragmatic risk stratification tool that would allow for the early identification of patients with COVID-19 infection who are at the highest risk of death could help guide the management of individual patients as well as resource utilization.
In a systematic review from April 2020, Wynants et al found that prediction models have rapidly entered the literature since the start of the COVID-19 pandemic, but that they are of questionable quality and at high risk of bias. As such, they are not ready for general use. A prediction model based on a large cohort with high quality methods would be of great value to the medical community. This BMJ article “Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO Clinical Characterisation Protocol: development and validation of the 4C Mortality Score” from August 2020 may be the prediction model we have been seeking.
The Paper
This is an ongoing prospective observational cohort study by the ISARIC Coronavirus Clinical Characterisation Consortium (ISARIC-4C) in 260 hospitals across England, Scotland and Wales. Their goal was to develop and validate a risk score to predict mortality in patients admitted to the hospital with COVID-19. They performed the study on two cohorts of patients: the derivation cohort from Feb. 6, 2020 to May 20, 2020 and the validation cohort from May 21 to June 29, 2020.
They chose potential predictor variables to be included in the score based on three criteria: patient and clinical variables known to influence outcome in pneumonia and flu-like illness, clinical biomarkers previously identified within the literature as potential predictors in patients with COVID-19, values available for at least two-thirds of patients within the cohort. The authors identified existing risk stratification scores using a systematic literature search and compared their new 4C score to these existing scores using an area under the receiver operating characteristic curve statistic for comparison.
Researchers enrolled consecutive patients 18 years or older who were admitted to the hospital with COVID-19 infection at least four weeks and any patient admitted within four weeks of data extraction were excluded. The primary endpoint assessed was in-hospital mortality. The derivation dataset included 35,463 patients. The median age of this patient set was 73; 41.7% of patients were female and 76.0% of patients had at least one comorbidity. In this dataset, they found a mortality rate of 32.2%. The validation dataset included 22,361 patients and had a mortality rate of 30.1%.
For model creation, 41 candidate predictor variables were measured and after composite variable creation, 21 variables remained. Of these, eight important predictors were identified to formulate the 4C Mortality Score: age, sex, number of comorbidities, respiratory rate, peripheral oxygen saturation, Glasgow coma scale, urea level, C reactive protein. Researchers chose to give each comorbidity equal weight as previous evidence suggested an additive effect of comorbidity in patients with COVID-19 and an increased number of comorbidities was associated with poorer clinical outcomes. (Guan 2020) The score has a range of 0-21 points.
The 4C Mortality Score showed high discrimination for mortality. For the derivation cohort, the area under the receiver operating characteristic curve was 0.79, with a 95% confidence interval 0.78 to 0.79. The validation cohort had area under the receiver operating characteristic curve 0.77 with a 95% confidence interval 0.76 to 0.77. Patients in the very high risk group (score ≥ 15) were found to have a mortality rate of 61.5% and the score had a positive predictive value of 61.5% while patients in the low risk group (score ≤ 3) had a mortality rate of 1.2% and the score had a negative predictive value of 98.8%.
| Risk Groups | 4C Score | % Mortality | Predictive Value |
| Very High Risk | ≥ 15 | 61.5% | PPV 61.5% |
| High Risk | 9-14 | 31.4% | NPV 68.6% |
| Intermediate Risk | 4-8 | 9.9% | NPV 90.1% |
| Low Risk | ≤ 3 | 1.2% | NPV 98.8% |
The 4C Mortality Score performed very well when compared to existing scores. The researchers compared the compared discriminatory performance of the 4C score with 15 other pre-existing risk stratification scores and found the 4C score had an area under the curve (AUC) 0.61 to 0.76. While other COVID-19 scores had AUC of 0.63 to 0.73.
There are a number of strengths to this study. First, it asks a clinically important question as early mortality prediction can guide management and resource allocation. Second, the 4C Mortality Score uses patient demographics, clinical observations and blood parameters that are routinely collected and available at the initial presentation, allowing for a prediction score to be generated early in the patient’s course.
The score is made even more clinically applicable because the researchers performed temporal and geographical validation to account for changes in patient characteristics and baseline infection rates — factors that can be expected to change throughout the course of a pandemic.
Third, the study population is large and the researchers used all available data to maximize the power and generalizability of the results. Fourth, the model reliability was assessed using a temporally distinct validation cohort with sensitivity analyses. Researchers restricted analysis outcomes to patients who were admitted ≥4 weeks before final data extraction to enable most patients to complete their hospital admission. And last, the authors provide a transparent and thorough explanation of variables included in the derivation of the 4C model.
As with even well done studies, there are several limitations. First, the list of included comorbidities was not exhaustive and excluded some potentially relevant conditions, including hypertension, previous myocardial infarction and stroke. Second, the 4C Morbidity Score was not compared to existing scores that required a large number of parameters (APACHE II & scores requiring advanced imaging modalities). While this makes the 4C Morbidity Score more clinically applicable, its value compared to these scores cannot be determined in this paper.
Third, patients included were those admitted to the hospital, having a higher mortality and more advanced age than the general community population with COVID-19 infection. The score is for inpatient use and cannot be applied in the outpatient setting and potentially cannot be generalized to younger patients. Next, the study included a predominantly Caucasian population in a non-low income healthcare system. These results may not extrapolate to lower income countries where preventative medicine is not as robust. And last, the study patient population was older (age ≈73 years), making the applicability to younger populations uncertain until further validation in this cohort of patients occurs.
While the researchers performed a literature search to identify existing risk scores with which to compare the 4C score, there are a few factors regarding this comparison worth discussion. The discriminatory performance of existing COVID-19 scores applied to this cohort of patients was lower than reported in derivation. This is most likely due to use of small inpatient cohorts resulting in generous results. Additionally, there were seven risk scores that the authors could apply in less than 10% of their validation cohort. This is because these scores required a lactate dehydrogenase measurement, which is not a routinely measured laboratory value in the United Kingdom.
Bottom Line
This risk-stratification score to predict risk of mortality for patients with COVID-19 infection was developed using a large COVID-19 dataset and has been thoroughly internally validated.
The patient population enrolled, however, will limit the widespread applicability at this time. Once externally validated, this tool could prove useful in guiding resource allocation and determining disposition for COVID-19 patients. If you are looking for a prediction tool in the appropriate patient population, an online calculator can be found at: https://isaric4c.net/risk/.
References
Guan WJ, Liang WH, Zhao Y, Liang HR, Chen ZS, Li YM, Liu XQ, Chen RC, Tang CL, Wang T, Ou CQ, Li L, Chen PY, Sang L, Wang W, Li JF, Li CC, Ou LM, Cheng B, Xiong S, Ni ZY, Xiang J, Hu Y, Liu L, Shan H, Lei CL, Peng YX, Wei L, Liu Y, Hu YH, Peng P, Wang JM, Liu JY, Chen Z, Li G, Zheng ZJ, Qiu SQ, Luo J, Ye CJ, Zhu SY, Cheng LL, Ye F, Li SY, Zheng JP, Zhang NF, Zhong NS, He JX; China Medical Treatment Expert Group for COVID-19. Comorbidity and its impact on 1590 patients with COVID-19 in China: a nationwide analysis. Eur Respir J. 2020 May 14;55(5):2000547. doi: 10.1183/13993003.00547-2020. PMID: 32217650; PMCID: PMC7098485.
Knight SR, Ho A, Pius R, Buchan I, Carson G, Drake TM, Dunning J, Fairfield CJ, Gamble C, Green CA, Gupta R, Halpin S, Hardwick HE, Holden KA, Horby PW, Jackson C, Mclean KA, Merson L, Nguyen-Van-Tam JS, Norman L, Noursadeghi M, Olliaro PL, Pritchard MG, Russell CD, Shaw CA, Sheikh A, Solomon T, Sudlow C, Swann OV, Turtle LC, Openshaw PJ, Baillie JK, Semple MG, Docherty AB, Harrison EM; ISARIC4C investigators. Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO Clinical Characterisation Protocol: development and validation of the 4C Mortality Score. BMJ. 2020 Sep 9;370:m3339. doi: 10.1136/bmj.m3339. Erratum in: BMJ. 2020 Nov 13;371:m4334. PMID: 32907855.
