Assessing the Performance of Critical Care Outreach Triggers and Severity Scores in Predicting Patient Outcomes.

Early recognition of clinical deterioration in hospitalized patients is a fundamental component of safe and effective acute care. Physiological deterioration is frequently preceded by measurable abnormalities in routinely monitored parameters, providing an opportunity for timely recognition and intervention before progression to serious adverse events. Failure to identify and respond to these warning signs may result in preventable complications, including cardiac arrest, unplanned intensive care unit (ICU) admission, and in-hospital mortality.

 

The concept of identifying hospitalized patients at risk of deterioration and providing timely escalation of care has evolved substantially over the past two decades. Critical care outreach was developed to extend critical care expertise beyond the ICU to patients in general hospital wards who may require higher levels of monitoring or organ support. Riley and Faleiro described the rationale for critical care outreach and emphasized the importance of objective clinical criteria for identifying patients at risk of deterioration, together with early intervention by Critical Care Outreach Teams (CCOTs).¹ The CCOT risk-scoring approach described in this context provided a structured framework for identifying physiological deterioration and determining the need for critical care review. In the present study, the CCOT score was adapted from the scoring criteria described by Riley and Faleiro and was used as one of the clinical risk-stratification tools.¹.

 

The development of early warning systems subsequently provided a more standardized approach to recognizing physiological deterioration. Subbe et al. validated the Modified Early Warning Score (MEWS), demonstrating an association between increasing physiological derangement and adverse outcomes, including ICU admission and mortality. ² MEWS incorporates routinely available physiological variables and offers a simple bedside method for identifying patients who may require escalation of care. Its relative simplicity has contributed to its widespread adoption, particularly in settings where continuous electronic monitoring or sophisticated predictive systems may not be readily available.

 

In parallel, structured triage systems were developed to assist with assessment of acute illness severity. The Emergency Severity Index (ESI), a five-level triage system, incorporates both clinical acuity and anticipated resource requirements. Wuerz et al. demonstrated the reliability and validity of the ESI in emergency care settings. ³ Although ESI was primarily designed for emergency department triage rather than inpatient deterioration, its incorporation of clinical judgment and anticipated resource utilization provides a broader assessment of acuity than purely physiology-based early warning scores.

 

The implementation of structured critical care outreach and early warning systems has subsequently been evaluated in clinical practice. Priestley et al. reported the results of a ward-randomized trial evaluating the phased introduction of critical care outreach and demonstrated the potential benefits of an outreach model in the management of deteriorating ward patients.⁴ These findings contributed to the growing recognition that early identification alone is insufficient unless it is linked to an effective system for assessment, escalation, and intervention.

 

The concept subsequently evolved into broader Rapid Response Systems (RRS), incorporating mechanisms for detection of deterioration, activation of an appropriately skilled response team, and delivery of timely intervention. Calzavacca et al. demonstrated that delays between the development of trigger criteria and activation of a medical emergency team were clinically relevant and that maturation of an RRS was associated with changes in activation patterns and outcomes.⁵ Jones et al. subsequently reviewed the development and evidence surrounding rapid-response teams, highlighting their role in responding to hospitalized patients with acute clinical deterioration.⁶ A systematic review by Maharaj et al. further demonstrated that RRS implementation may reduce adverse outcomes such as cardiac arrest and hospital mortality, although the magnitude of benefit varies according to system design, implementation, staffing, and institutional context.⁷

 

The need for standardized physiological assessment led to the development of the National Early Warning Score (NEWS), followed by the updated NEWS2 system. The Royal College of Physicians introduced NEWS2 as a standardized approach to assessing acute illness severity, incorporating respiratory rate, oxygen saturation, supplemental oxygen requirement, temperature, systolic blood pressure, heart rate, and level of consciousness.⁸ Subsequent studies have evaluated the performance of NEWS and NEWS2 in hospitalized patients, with evidence suggesting improved identification of patients at risk of deterioration compared with some earlier scoring approaches, although performance varies across patient populations and clinical settings.⁹⁻¹¹

 

Despite the increasing use of early warning systems, important limitations remain. Traditional scores generally provide a snapshot of physiological status and may not fully capture the complexity, trajectory, or clinical context of individual patients. Furthermore, an association between increasing score and adverse outcome does not necessarily translate into strong individual-level discrimination. Consequently, considerable overlap may exist between the scores of patients who subsequently deteriorate and those who do not. This limitation is particularly relevant when scores are used to guide escalation decisions at the individual patient level.

 

System-level factors may also influence the effectiveness of deterioration-detection systems. The timing of recognition and activation, availability of trained personnel, response time, staffing patterns, and differences between routine and out-of-hours care may all affect patient outcomes. Evidence examining out-of-hours critical care processes has highlighted the importance of timing and organizational factors in determining outcomes. ¹² These observations suggest that the performance of a scoring system cannot be considered independently of the clinical and operational system in which it is implemented.

 

More recent work has focused on comparing and refining early warning systems. Pimentel et al. evaluated NEWS and NEWS2 for predicting clinical deterioration in hospitalized patients, while subsequent studies have further examined the performance of NEWS2 across acute hospital populations. ¹³⁻¹⁵ Although these systems provide standardized and reproducible assessments, their performance remains dependent on the patient population, clinical context, and outcome being predicted.

 

At the same time, advances in clinical informatics and machine learning have generated interest in dynamic prediction models that integrate large numbers of physiological, laboratory, and temporal variables. Such approaches may improve prediction of clinical deterioration by identifying complex patterns that are not captured by conventional bedside scores.¹⁶˒¹⁷ However, issues relating to data availability, interpretability, external validation, workflow integration, and clinical implementation remain important barriers to widespread adoption. Conventional bedside scores therefore continue to have an important role because of their simplicity, transparency, and ease of application.

 

In this context, there remains a need to evaluate commonly used clinical scoring systems in real-world populations undergoing CCOT assessment, particularly in healthcare settings where local patient characteristics, staffing structures, and resource availability may differ from those in which these scores were originally developed or validated. Direct comparison of MEWS, CCOT score, and ESI may provide insight into whether a physiology-based score, a CCOT-specific risk score, or a broader acuity and resource-based triage system offers greater utility in predicting clinically important outcomes.

 

The present study therefore aims to evaluate and compare the predictive performance of the Modified Early Warning Score (MEWS), CCOT score, and Emergency Severity Index (ESI) among hospitalized adult patients requiring CCOT activation in a tertiary-care setting. The primary objective is to assess their association and discriminatory performance with respect to clinically relevant outcomes, including in-hospital mortality, ICU admission/ICU stay, and total hospital length of stay. The study also evaluates the influence of the timing of CCOT activation and departmental variation on patient outcomes. By comparing these tools within the same patient population and clinical environment, this study seeks to determine their relative utility for risk stratification and to identify potential limitations in their application to deteriorating hospitalized patients.

 

We hypothesize that MEWS and the CCOT score will demonstrate significant associations with adverse clinical outcomes because of their ability to quantify physiological deterioration, but that their individual-level discriminative performance may be limited. We further hypothesize that ESI, through incorporation of broader measures of acuity and anticipated resource requirements, may provide complementary or improved predictive information. The findings of this study may contribute to the optimization of CCOT activation strategies and support the development of more effective, context-specific approaches to the recognition and management of hospitalized patients at risk of clinical deterioration.

A total of 208 adult inpatients with CCOT activation were included in the study. Of these, 188 patients (90.4%) survived and were discharged, while 20 patients (9.6%) either expired or left against medical advice (LAMA) (Figure 1).

Patients with adverse outcomes demonstrated significantly higher illness severity scores, including MEWS (7.0 ± 4.2 vs 3.6 ± 1.8; p < 0.001) and CCOT score (12.2 ± 5.8 vs 6.4 ± 2.1; p < 0.001), along with significantly lower ESI scores (1.9 ± 0.8 vs 2.7 ± 0.6; p < 0.001), compared with survivors.

 

ROC curve analysis demonstrated that ESI had the highest predictive accuracy for in-hospital outcome, with an AUC of 0.76, indicating good discriminatory performance. In comparison, MEWS and CCOT scores showed poor predictive ability, with AUCs of 0.24 and 0.20, respectively, whereas the composite model combining all scores showed no improvement (AUC = 0.21) (Figure 2)

 

Correlation analysis revealed no significant association between any of the scoring systems and ICU stay duration or length of hospital stay (LOHS). However, significant correlations were observed with patient outcomes, with CCOT score showing the strongest inverse correlation (r = –0.542, p < 0.00001), followed by MEWS (r = –0.426, p < 0.00001). In contrast, ESI demonstrated a moderate positive correlation with survival outcome (r = 0.347, p < 0.00001). Comparison based on timing of CCOT activation showed that patients assessed after 4 PM had significantly higher MEWS and CCOT scores and lower ESI scores, indicating greater physiological derangement; however, discharge rates between before-4 PM and after-4 PM groups were not significantly different (92% vs 86%, p = 0.167). ICU stay duration was also comparable between the groups, although a small but statistically significant difference was observed in hospital stay duration (p = 0.019) (Table 1).

 

 

Table 1 - Comparison of CCOT activations before and after 4 PM

 

Table 2 – Correlation between MEWS, CCOT, and ESI with Length of ICU stay, Length of Hospital stay (LOHS) and outcomes.

 

 

Figure 1a – Modified Early Warning Score (MEWS)

Figure 1b – Emergency Severity Index (ESI)

 

 

Figure 1c- Critical Care Outreach Team (CCOT) scoring system

 

Figure 3

 

 

 

Figure 4 - The ROC curve analysis between ESI, CCOT, and MEWS.

Critical Care Outreach Teams (CCOTs) and rapid response systems have evolved as important components of hospital-based strategies for the early recognition and management of clinical deterioration. Their primary objective is to identify patients at risk of further physiological deterioration and facilitate timely assessment, escalation, and intervention before progression to major adverse events. Critical care outreach developed in response to the recognition that critical care expertise could extend beyond the ICU to deteriorating ward patients through structured clinical assessment and predefined trigger criteria.¹ Subsequent studies evaluating critical care outreach and rapid response systems have suggested potential reductions in adverse outcomes, although the magnitude of benefit has varied according to patient population, system organization, staffing, and timeliness of response.²⁻⁵

 

The present study evaluated the prognostic performance of three clinically different assessment tools—Modified Early Warning Score (MEWS), CCOT score, and Emergency Severity Index (ESI)—among hospitalized adult patients undergoing CCOT activation. The principal finding was that ESI showed better discrimination for in-hospital mortality than either MEWS or the CCOT score, with an area under the receiver operating characteristic curve (AUC) of 0.76. Although patients with adverse outcomes demonstrated greater physiological derangement reflected by higher MEWS and CCOT scores and lower ESI scores, the relatively poor discriminatory performance of MEWS and CCOT indicates that association with clinical severity does not necessarily translate into adequate individual-level prediction of mortality.

 

Performance of MEWS and CCOT scores

MEWS was developed as a simple bedside tool to identify patients with physiological abnormalities who may be at increased risk of deterioration. Subbe et al. demonstrated that increasing MEWS was associated with ICU admission and mortality, establishing its utility as an early warning tool in hospitalized medical patients. ⁶ The subsequent development of NEWS and NEWS2 represented efforts to improve standardization and sensitivity for detecting deterioration by incorporating additional physiological variables, including oxygen saturation and supplemental oxygen requirement. ⁷⁻¹⁰

 

In the present study, higher MEWS values were observed among patients experiencing adverse outcomes, confirming that greater physiological derangement was associated with poorer clinical outcomes. However, the ROC analysis demonstrated poor discrimination. This finding is clinically important because an early warning score may perform well as a surveillance and trigger tool without necessarily functioning as a strong prognostic model. The purpose of MEWS is primarily to identify physiological instability and prompt clinical reassessment rather than to provide an individualized estimate of mortality risk.

 

The relatively poor discriminatory performance observed in our cohort may also reflect the characteristics of patients referred to CCOT. Once a patient has fulfilled criteria for outreach activation, the cohort is already enriched for physiological instability and clinical concern. Consequently, the range of MEWS values may become compressed, reducing the ability of the score to distinguish between survivors and non-survivors. In addition, ward patients requiring CCOT intervention represent a heterogeneous population in whom the underlying cause, trajectory of illness, comorbidity burden, treatment limitations, and timing of escalation may influence outcome independently of the initial physiological score.

 

The CCOT score showed a similar pattern. Patients with adverse outcomes had higher CCOT scores, indicating greater physiological abnormality and clinical risk. This is consistent with the rationale underlying critical care outreach, in which objective clinical criteria are used to identify patients who may require escalation of care. ¹ However, the poor discriminatory performance of the CCOT score suggests that the score may be more useful for identifying patients requiring outreach assessment than for accurately predicting subsequent mortality.

 

An additional consideration is that the CCOT score used in this study was based on criteria described in the early development of critical care outreach and adapted to the local clinical context. ¹ Unlike extensively validated scores such as NEWS2, locally implemented CCOT scores may vary in the variables included, thresholds applied, and escalation pathways attached to particular scores. Therefore, their prognostic performance may differ substantially between institutions. External validation and prospective assessment would be required before such a score could be generalized to other healthcare settings.

 

Superior discriminatory performance of ESI

The most notable finding of the study was the superior performance of ESI, which demonstrated an AUC of 0.76 for predicting in-hospital mortality. ESI was originally developed as a five-level emergency department triage system intended to classify patients according to clinical acuity and anticipated resource requirements rather than to function as a mortality prediction model.¹¹ Its performance in the present CCOT population is therefore particularly interesting.

 

The lower ESI categories observed among patients with adverse outcomes suggest that patients judged to have greater acuity and higher anticipated resource requirements were more likely to experience poor outcomes. Unlike MEWS, which is predominantly based on physiological measurements, ESI incorporates an assessment of overall clinical acuity and anticipated resource utilization. This multidimensional approach may have provided additional prognostic information in the heterogeneous population undergoing CCOT activation.

 

However, the present findings should not be interpreted as demonstrating that ESI is intrinsically superior to MEWS or NEWS2 for predicting deterioration in hospitalized patients. The ESI was developed for a different clinical environment, and its performance in this study may partly reflect the clinical characteristics and workflow of the local CCOT population. In addition, ESI incorporates clinical judgment, which may capture information not represented by routinely measured physiological variables. This may be particularly relevant in patients whose risk is influenced by factors such as rapidly evolving clinical conditions, anticipated need for organ support, or overall clinical appearance.

 

The findings nevertheless raise the possibility that broader assessments of acuity may complement conventional physiological early warning scores in patients already identified as requiring critical care outreach. Further prospective studies comparing ESI with MEWS, NEWS2, and validated deterioration models in CCOT populations would be necessary to determine whether this observation is reproducible.

 

Timing of CCOT activation and after-hours deterioration

Another important finding was the difference in clinical severity according to the timing of CCOT activation. Patients activated after 4 PM had significantly higher MEWS and CCOT scores and lower ESI scores, suggesting greater physiological derangement among patients requiring outreach assessment outside routine working hours. Although the difference in mortality between patients activated before and after 4 PM did not reach statistical significance, the observed trend toward higher mortality during the after-hours period is clinically relevant.

 

The timing of recognition and escalation is an important component of rapid response system effectiveness. Rapid response systems are dependent not only on the ability to identify deterioration but also on the availability of an appropriately skilled team and the speed with which the response is initiated.⁴˒⁵ Previous literature has emphasized that delays in recognition and escalation may influence outcomes, particularly during periods of reduced staffing or altered patterns of clinical surveillance.¹²˒¹³

 

Our finding of greater physiological derangement among after-hours CCOT activations may have several possible explanations. Patients may deteriorate later in their clinical course, recognition may be delayed until routine staffing levels decrease, or changes in ward surveillance and escalation practices may influence the timing of referral. Alternatively, the difference may reflect differences in the underlying case mix between daytime and after-hours referrals. Because the mortality difference did not reach statistical significance, these findings should be interpreted as hypothesis-generating rather than evidence of a definitive after-hours mortality effect.

 

Nevertheless, the observation supports the importance of maintaining consistent mechanisms for recognition, activation, and response throughout the 24 hours. A CCOT system that is available only during selected hours may fail to provide equivalent opportunities for early intervention across the hospital day.

 

Relationship with ICU and hospital length of stay

In the present study, the absence of significant correlations between the evaluated scoring systems and ICU or total hospital length of stay suggests that these tools may have greater utility for identifying immediate clinical risk than for predicting subsequent resource utilization or duration of hospitalization.

 

Length of stay is inherently multifactorial and is influenced by factors that extend beyond acute physiological severity. These include baseline comorbidity, functional status, complications during hospitalization, response to treatment, rehabilitation requirements, availability of step-down facilities, discharge planning, and institutional practices. Consequently, a score measured at the time of CCOT activation may reasonably be expected to have limited ability to explain the eventual duration of hospitalization.

 

Similarly, ICU length of stay is influenced by the trajectory of organ dysfunction after admission, response to treatment, development of complications, and decisions regarding escalation or de-escalation of care. Therefore, the absence of a significant relationship between baseline early warning scores and ICU or hospital length of stay does not necessarily indicate that these scores lack clinical utility. Rather, it reinforces the distinction between prediction of acute deterioration or mortality and prediction of downstream healthcare utilization.

 

Combined scoring models

An additional finding was that combining MEWS and CCOT score with ESI did not improve predictive performance. The composite approach demonstrated poor discrimination despite ESI alone showing an AUC of 0.76. This suggests that the information provided by the three scores may not be complementary in the manner anticipated

 

One possible explanation is substantial overlap between the variables captured by MEWS and CCOT score, both of which assess physiological deterioration. Adding correlated measures may therefore provide limited incremental information. Furthermore, the incorporation of ESI into a simple composite score may introduce differences in scale, weighting, and clinical constructs that are not appropriately handled by unweighted combination. A score designed for triage may capture dimensions of clinical acuity that cannot simply be added to physiological scores without formal model development and calibration.

 

This finding highlights an important principle in clinical prediction research: adding more scores does not necessarily improve prediction. Incremental predictive value should be assessed using appropriately developed multivariable models, discrimination metrics, calibration, and measures of clinical utility rather than by simple aggregation of existing scores.

 

Implications for contemporary deterioration prediction

The limitations of conventional early warning scores have generated increasing interest in dynamic prediction models and machine learning. These approaches can integrate multiple physiological measurements, laboratory variables, temporal trends, and clinical characteristics to generate individualized estimates of deterioration risk. ¹⁴˒¹⁵ Such models may potentially identify subtle changes in patient trajectory that are not captured by a single cross-sectional score.

 

However, the complexity of machine learning models also introduces challenges related to interpretability, data quality, external validation, workflow integration, and generalizability. Therefore, despite their potential advantages, they should not be considered a replacement for simple bedside assessment without robust prospective validation.

 

The findings of the present study reinforce the continued importance of conventional scores while demonstrating that their clinical roles should be appropriately defined. MEWS and CCOT scores may be particularly valuable as structured trigger and surveillance tools, whereas ESI may provide additional information regarding overall acuity in selected CCOT populations. None of these scores, however, should be considered a substitute for clinical assessment and reassessment of a deteriorating patient.

 

Strengths And Limitations

This study provides a direct comparison of three clinically distinct scoring approaches within the same population of hospitalized patients undergoing CCOT activation. The inclusion of MEWS, CCOT score, and ESI allows comparison between physiological deterioration assessment, outreach-specific criteria, and broader acuity-based triage. Evaluation of timing of CCOT activation also provides insight into an important operational component of outreach services.

 

Several limitations should be acknowledged. First, this was a single-center observational study, and the relatively small sample size may limit statistical power and generalizability. Second, the observational design precludes causal conclusions regarding the relationship between score values, timing of activation, and clinical outcomes. Third, the CCOT score was locally adapted from the criteria described in the early critical care outreach literature and has not undergone extensive external validation. Fourth, ESI was originally designed for emergency department triage and not specifically for mortality prediction among hospitalized CCOT patients; therefore, its observed performance should be interpreted within the context of this study population. Finally, factors such as comorbidities, treatment limitations, time from physiological deterioration to CCOT activation, and subsequent interventions may have influenced outcomes but were not necessarily captured by the scoring systems.

In this study of hospitalized adults undergoing CCOT activation, ESI demonstrated better discriminatory performance for in-hospital mortality than MEWS and CCOT score, with an AUC of 0.76. Higher MEWS and CCOT scores and lower ESI scores were associated with adverse outcomes, indicating that all three tools reflected clinical severity to some extent; however, MEWS and CCOT demonstrated poor discriminatory ability for individual outcome prediction. Patients activated after 4 PM had greater physiological derangement, although the observed increase in mortality did not reach statistical significance.

 

These findings suggest that conventional physiological scores may be more appropriate as early recognition and escalation tools than as standalone mortality prediction models. The superior performance of ESI in this cohort warrants further investigation but should be interpreted cautiously given its original purpose as an emergency department triage instrument. Future multicentre prospective studies with larger cohorts and formal model validation are required to determine whether ESI or other multidimensional approaches can improve risk stratification in CCOT populations. Integration of clinical scoring with dynamic physiological and laboratory data, including appropriately validated machine-learning approaches, may ultimately provide more accurate individualized prediction of deterioration and support more timely escalation of care.

1.       Riley B, Faleiro R. Critical care outreach: rationale and development. BJA CEPD Reviews. 2001;1(5):146–149. doi:10.1093/bjacepd/1.5.146.

2.       Subbe CP, Kruger M, Rutherford P, Gemmel L. Validation of a modified Early Warning Score in medical admissions. QJM. 2001;94:521–526.

3.       Wuerz RC, Milne LW, Eitel DR, Travers D, Gilboy N. Reliability and validity of a new five-level triage instrument. Acad Emerg Med. 2000;7:236–242.

4.       Priestley G, Watson W, Rashidian A, et al. Introducing critical care outreach: a ward-randomised trial of phased introduction in a general hospital. Lancet. 2004;364:343–349.

5.       Calzavacca P, Licari E, Tee A, et al. The impact of Rapid Response System on delayed emergency team activation patient characteristics and outcomes—a follow-up study. Resuscitation. 2010;81(1):31–35. doi:10.1016/j.resuscitation.2009.09.026.

6.       Jones DA, DeVita MA, Bellomo R. Rapid-response teams. N Engl J Med. 2011;365:139–146.

7.       Maharaj R, Raffaele I, Wendon J. Rapid response systems: a systematic review and meta-analysis. Crit Care. 2015;19:254.

8.       Royal College of Physicians. National Early Warning Score (NEWS) 2: Standardising the assessment of acute-illness severity in the NHS. London: Royal College of Physicians; 2017.

9.       Vollam S, Dutton S, Lamb S, et al. Out-of-hours discharge from intensive care, in-hospital mortality and intensive care readmission rates: a systematic review and meta-analysis. Intensive Care Med. 2018;44:1115–1129.

10.    Pimentel MAF, Redfern OC, Gerry S, et al. Comparison of the ability of the National Early Warning Score and National Early Warning Score 2 to discriminate clinical deterioration in hospitalized patients. Resuscitation. 2019;134:147–156.

11.    Kostakis I, Smith GB, Prytherch D, et al. The performance of the National Early Warning Score and NEWS2 in hospitalized patients. Resuscitation. 2021;159:150–157.

12.    Scott LJ, Redmond NM, Tavaré A, et al. Validation of NEWS2 in acute hospital settings. BMJ Open. 2021;11.

13.    Aneman A, Parr MJA. Timing and effectiveness of rapid response systems. Crit Care Resusc. 2021.

14.    Shamout FE, Zhu T, Clifton DA. Machine learning for clinical deterioration prediction. NPJ Digit Med. 2021.

15.    Barton C, Chettipally U, Zhou Y, et al. Early detection of patient deterioration using machine learning. JAMA Netw Open. 2022.