The clinical usefulness of prognostic prediction models in critical illness

  • Tim Baker
    Corresponding author at: Global Health—Health Systems & Policy, Department of Public Health Sciences, Karolinska Institutet, Stockholm, Sweden.
    Global Health—Health Systems & Policy, Department of Public Health Sciences, Karolinska Institutet, Stockholm, Sweden

    Perioperative Medicine & Intensive Care, Karolinska University Hospital, Stockholm, Sweden

    Department of Anaesthesia & Intensive Care, Queen Elizabeth Central Hospital, Blantyre, Malawi
    Search for articles by this author
  • Martin Gerdin
    Global Health—Health Systems & Policy, Department of Public Health Sciences, Karolinska Institutet, Stockholm, Sweden

    Department of Clinical Science, Innovation, and Technology, Karolinska Institutet, 171 77 Stockholm, Sweden
    Search for articles by this author
Published:September 19, 2017DOI:


      • Prognostic prediction models are widespread within acute and critical care.
      • To be clinically useful they should have a good predictive performance, be user-friendly and guide important decisions.
      • Intervention research is required to evaluate their impact on patient outcomes.


      Critical illness is any immediately life-threatening disease or trauma and results in several million deaths globally every year. Responsive hospital systems for managing critical illness include quick and accurate identification of the critically ill patients. Prognostic prediction models are widely used for this aim. To be clinically useful, a model should have good predictive performance, often measured using discrimination and calibration. This is not sufficient though: a model also needs to be tested in the setting where it will be used, it should be user-friendly and should guide decision making and actions. The clinical usefulness and impact on patient outcomes of prediction models has not been greatly studied. The focus of research should shift from attempts to optimise the precision of models to real-world intervention studies to compare the performance of models and their impacts on outcomes.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to European Journal of Internal Medicine
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Baker T.
        Critical care in low resource settings.
        Stockholm, Karolinska Institutet2015
        • Adhikari N.K.
        • Fowler R.A.
        • Bhagwanjee S.
        • Rubenfeld G.D.
        Critical care and the global burden of critical illness in adults.
        Lancet. 2010; 376: 1339-1346
        • GBD_2013
        Global, regional, and national age-sex specific all-cause and cause-specific mortality for 240 causes of death, 1990–2013: a systematic analysis for the Global Burden of Disease Study 2013.
        Lancet. 2014;
        • Knaus W.A.
        • Wagner D.P.
        • Draper E.A.
        • Zimmerman J.E.
        • Bergner M.
        • Bastos P.G.
        • et al.
        The APACHE III prognostic system: risk prediction of hospital mortality for critically III hospitalized adults.
        Chest. 1991; 100: 1619-1636
        • Knaus W.A.
        • Draper E.A.
        • Wagner D.P.
        • Zimmerman J.E.
        Apache II—a severity of disease classification system.
        Crit Care Med. 1985; 13
      1. APACHE—acute physiology and chronic health evaluation: a physiologically based classification system. 1981
        • Higgins T.L.
        • Teres D.
        • Copes W.S.
        • Nathanson B.H.
        • Stark M.
        • Kramer A.A.
        Assessing contemporary intensive care unit outcome: an updated Mortality Probability Admission Model (MPM0-III).
        Crit Care Med. 2007; 35: 827-835
        • Le Gall J.-R.
        • Lemeshow S.
        • Saulnier F.
        A new Simplified Acute Physiology Score (SAPS II) based on a European/North American Multicenter Study.
        JAMA. 1993; 270: 2957
        • Le Gall J.-R.
        • Loirat P.
        • Annick A.
        • Glaser P.
        • Granthil C.
        • Mathieu D.
        • et al.
        A simplified acute physiology score for ICU patients.
        Crit Care Med. 1984; 12
        • Lemeshow S.
        • Teres D.
        • Klar J.
        • Avrunin J.S.
        • Gehlbach S.H.
        • Rapoport J.
        Mortality Probability Models (MPM II) based on an international cohort of intensive care unit patients.
        JAMA. 1993; 270: 2478-2486
        • Lemeshow S.
        • Teres D.
        • Pastides H.
        • Avrunin J.S.
        • Steingrub J.S.
        A method for predicting survival and mortality of ICU patients using objectively derived weights.
        Crit Care Med. 1985; 13: 519-525
        • Zimmerman J.E.
        • Kramer A.A.
        • McNair D.S.
        • Malila F.M.
        Acute physiology and chronic health evaluation (APACHE) IV: hospital mortality assessment for today's critically ill patients.
        Crit Care Med. 2006; 34: 1297-1310
        • Moreno R.P.
        • Metnitz P.G.H.
        • Almeida E.
        • Jordan B.
        • Bauer P.
        • Campos R.A.
        • et al.
        SAPS 3—from evaluation of the patient to evaluation of the intensive care unit. Part 2: development of a prognostic model for hospital mortality at ICU admission.
        Intensive Care Med. 2005; 31: 1345-1355
        • Bell M.B.
        • Konrad D.
        • Granath F.
        • Ekbom A.
        • Martling C.R.
        Prevalence and sensitivity of MET-criteria in a Scandinavian University Hospital.
        Resuscitation. 2006; 70: 66-73
        • Baker T.
        • Blixt J.
        • Lugazia E.
        • Schell C.O.
        • Mulungu M.
        • Milton A.
        • et al.
        Single deranged physiologic parameters are associated with mortality in a low-income country.
        Crit Care Med. 2015; 43: 2171-2179
        • Goldhill D.R.
        • White S.A.
        • Sumner A.
        Physiological values and procedures in the 24 h before ICU admission from the ward.
        1999: 529-534
        • Franklin C.
        • Mathew J.
        Developing strategies to prevent inhospital cardiac arrest: analyzing responses of physicians and nurses in the hours before the event.
        Crit Care Med. 1994; 22: 244-247
        • Bleyer A.J.
        • Vidya S.
        • Russell G.B.
        • Jones C.M.
        • Sujata L.
        • Daeihagh P.
        • et al.
        Longitudinal analysis of one million vital signs in patients in an academic medical center.
        Resuscitation. 2011; 82: 1387-1392
        • Rylance J.
        • Baker T.
        • Mushi E.
        • Mashaga D.
        Use of an early warning score and ability to walk predicts mortality in medical patients admitted to hospitals in Tanzania.
        Trans R Soc Trop Med Hyg. 2009; 103: 790-794
        • Smith G.B.
        • Prytherch D.R.
        • Schmidt P.E.
        • Featherstone P.I.
        Review and performance evaluation of aggregate weighted ‘track and trigger’ systems.
        Resuscitation. 2008; 77: 170-179
        • Prytherch D.R.
        • Smith G.B.
        • Schmidt P.E.
        • Featherstone P.I.
        ViEWS—towards a national early warning score for detecting adult inpatient deterioration.
        Resuscitation. 2010; 81: 932-937
        • Chapman S.M.
        • Wray J.
        • Oulton K.
        • Peters M.J.
        Systematic review of paediatric track and trigger systems for hospitalised children.
        Resuscitation. 2016; 109: 87-109
        • RCP
        National Early Warning Score (NEWS). Standardising the assessment of acute-illness severity in the NHS. Report of a working party.
        Royal College of Physicians, 2012
        • Seymour C.W.
        • Liu V.X.
        • Iwashyna T.J.
        • Brunkhorst F.M.
        • Rea T.D.
        • Scherag A.
        • et al.
        Assessment of clinical criteria for sepsis: for the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3).
        JAMA. 2016; 315: 762-774
        • Gerdin M.
        The risk of dying—predicting trauma mortality in urban Indian hospitals.
        Karolinska Institutet, Stockholm2015
        • Steyerberg E.W.
        • Vergouwe Y.
        Towards better clinical prediction models: seven steps for development and an ABCD for validation.
        Eur Heart J. 2014; 35: 1925-1931
        • Smith G.B.
        • Prytherch D.R.
        • Meredith P.
        • Schmidt P.E.
        • Featherstone P.I.
        The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death.
        Resuscitation. 2013; 84: 465-470
        • Kruisselbrink R.
        • Kwizera A.
        • Crowther M.
        • Fox-Robichaud A.
        • O'Shea T.
        • Nakibuuka J.
        • et al.
        Modified Early Warning Score (MEWS) identifies critical illness among ward patients in a resource restricted setting in Kampala, Uganda: a prospective observational study.
        PLoS One. 2016; 11e0151408
        • Wheeler I.
        • Price C.
        • Sitch A.
        • Banda P.
        • Kellett J.
        • Nyirenda M.
        • et al.
        Early warning scores generated in developed healthcare settings are not sufficient at predicting early mortality in Blantyre, Malawi: a prospective cohort study.
        PLoS One. 2013; 8e59830
        • Steyerberg E.W.
        • Vickers A.J.
        • Cook N.R.
        • Gerds T.
        • Gonen M.
        • Obuchowski N.
        • et al.
        Assessing the performance of prediction models: a framework for traditional and novel measures.
        Epidemiology. 2010; 21: 128-138
        • Gerdin M.
        • Roy N.
        • Fellander-Tsai L.
        • Tomson G.
        • von Schreeb J.
        • Petzold M.
        • et al.
        Traumatic transfers: calibration is adversely affected when prediction models are transferred between trauma care contexts in India and the United States.
        J Clin Epidemiol. 2016;
        • Vickers A.J.
        • Cronin A.M.
        Everything you always wanted to know about evaluating prediction models (but were too afraid to ask).
        Urology. 2010; 76: 1298-1301
        • Baker T.
        • Schell C.O.
        • Lugazia E.
        • Blixt J.
        • Mulungu M.
        • Castegren M.
        • et al.
        Vital signs directed therapy: improving care in an Intensive Care Unit in a low-income country.
        PLoS One. 2015; 10e0144801
        • Holmberg L.
        • Vickers A.
        Evaluation of prediction models for decision-making: beyond calibration and discrimination.
        PLoS Med. 2013; 10e1001491
        • Vickers A.J.
        • Elkin E.B.
        Decision curve analysis: a novel method for evaluating prediction models.
        Med Decis Mak. 2006; 26: 565-574
        • McGaughey J.
        • Alderdice F.
        • Fowler R.
        • Kapila A.
        • Mayhew A.
        • Moutray M.
        Outreach and Early Warning Systems (EWS) for the prevention of Intensive Care admission and death of critically ill adult patients on general hospital wards.
        Cochrane Database Syst Rev. 2007; 3
        • Hillman K.
        • Chen J.
        • Cretikos M.
        • Bellomo R.
        • Brown D.
        • Doig G.
        • et al.
        Introduction of the medical emergency team (MET) system: a cluster-randomised controlled trial.
        Lancet. 2005; 365: 2091-2097
        • Priestley G.
        • Mozley C.
        • Russell D.
        Introducing Critical Care Outreach: a ward-randomised trial of phased introduction in a general hospital.
        Intensive Care Med. 2004; 30: 1398-1404
        • Chan P.S.
        • Jain R.
        • Nallmothu B.K.
        • Berg R.A.
        • Sasson C.
        Rapid response teams: a systematic review and meta-analysis.
        Arch Intern Med. 2010; 170: 18-26
        • Winters B.D.
        • Weaver S.J.
        • Pfoh E.R.
        • Yang T.
        • Pham J.C.
        • Dy S.M.
        Rapid-response systems as a patient safety strategy: a systematic review.
        Ann Intern Med. 2013; 158: 417-425
        • Maharaj R.
        • Raffaele I.
        • Wendon J.
        Rapid response systems: a systematic review and meta-analysis.
        Crit Care (London, England). 2015; 19: 254
        • Alam N.
        • Hobbelink E.L.
        • Tienhoven A.J.V.
        • Ven P.M.V.D.
        • Jansma E.P.
        • Nanayakkara P.W.B.
        The impact of the use of the Early Warning Score (EWS) on patient outcomes: a systematic review.
        Resuscitation. 2014; 85: 587-594
        • Wagner A.K.
        • Soumerai S.B.
        • Zhang F.
        • Ross-degnan D.
        Segmented regression analysis of interrupted time series studies in medication use research.
        J Clin Pharm Ther. 2002; : 299-309