Development, Implementation, and Evaluation of an In-Hospital Optimized Early Warning Score for Patient Deterioration.


Background. Identification of patients at risk of deteriorating during their hospitalization is an important concern. However, many off-shelf scores have poor in-center performance. In this article, we report our experience developing, implementing, and evaluating an in-hospital score for deterioration. Methods. We abstracted 3 years of data (2014-2016) and identified patients on medical wards that died or were transferred to the intensive care unit. We developed a time-varying risk model and then implemented the model over a 10-week period to assess prospective predictive performance. We compared performance to our currently used tool, National Early Warning Score. In order to aid clinical decision making, we transformed the quantitative score into a three-level clinical decision support tool. Results. The developed risk score had an average area under the curve of 0.814 (95% confidence interval = 0.79-0.83) versus 0.740 (95% confidence interval = 0.72-0.76) for the National Early Warning Score. We found the proposed score was able to respond to acute clinical changes in patients' clinical status. Upon implementing the score, we were able to achieve the desired positive predictive value but needed to retune the thresholds to get the desired sensitivity. Discussion. This work illustrates the potential for academic medical centers to build, refine, and implement risk models that are targeted to their patient population and work flow.





Published Version (Please cite this version)


Publication Info

O'Brien, Cara, Benjamin A Goldstein, Yueqi Shen, Matthew Phelan, Curtis Lambert, Armando D Bedoya and Rebecca C Steorts (2020). Development, Implementation, and Evaluation of an In-Hospital Optimized Early Warning Score for Patient Deterioration. MDM policy & practice, 5(1). p. 2381468319899663. 10.1177/2381468319899663 Retrieved from

This is constructed from limited available data and may be imprecise. To cite this article, please review & use the official citation provided by the journal.



Benjamin Alan Goldstein

Professor of Biostatistics & Bioinformatics

I study the meaningful use of Electronic Health Records data. My research interests sit at the intersection of biostatistics, biomedical informatics, machine learning and epidemiology. I collaborate with researchers both locally at Duke as well as nationally. I am interested in speaking with any students, methodologistis or collaborators interested in EHR data.

Please find more information at:


Armando Diego Bedoya

Assistant Professor of Medicine

Unless otherwise indicated, scholarly articles published by Duke faculty members are made available here with a CC-BY-NC (Creative Commons Attribution Non-Commercial) license, as enabled by the Duke Open Access Policy. If you wish to use the materials in ways not already permitted under CC-BY-NC, please consult the copyright owner. Other materials are made available here through the author’s grant of a non-exclusive license to make their work openly accessible.