CONCERN Early Warning System Trial

@article{PhysioNet-concern-early-warning-1.0.0,
  author = {Rossetti, Sarah and Lau, Brandon and Cato, Kenrick},
  title = {{CONCERN Early Warning System Trial}},
  journal = {{PhysioNet}},
  year = {2026},
  month = oct,
  note = {Version 1.0.0},
  doi = {10.13026/qf6v-ev20},
  url = {https://doi.org/10.13026/qf6v-ev20}
}

@article{pollard_physionet_2026, title = {{PhysioNet} as a global platform for biomedical research}, volume = {1}, issn = {3005-0693}, url = {https://doi.org/10.1038/s44360-026-00096-z}, doi = {10.1038/s44360-026-00096-z}, abstract = {Twenty-five years after its first public repository was established, the PhysioNet resource for biomedical and clinical data has enabled tens of thousands of research papers. Further efforts are ongoing to advance secure data sharing, community engagement and scalable research infrastructure.}, number = {8}, journal = {Nature Health}, author = {Pollard, Tom and Moody, Benjamin E. and Lehman, Li-wei H. and Gow, Brian J. and Fernandes, Chrystinne and Xie, Chen and Johnson, Alistair and Mark, Roger G. and Heldt, Thomas}, month = aug, year = {2026}, pages = {792--795},}

Abstract

This dataset contains de-identified encounter level data from a 1-year, multisite, pragmatic, cluster-randomized controlled trial evaluating the CONCERN (COmmunicating Narrative Concerns Entered by RNs) early warning system, a machine learning-based tool that identifies patient deterioration risk from real-time nursing surveillance documentation patterns [1]. The trial randomized 74 inpatient clinical units (37 intervention, 37 usual care) across two U.S. health systems and included 60,893 adult hospital encounters (33,024 intervention, 27,869 usual care). Data cover coprimary outcomes (in-hospital mortality, length of stay) and secondary outcomes (cardiopulmonary arrest, sepsis, unanticipated ICU transfer, 30-day readmission), along with patient and encounter characteristics, unit assignment, and study arm. The dataset has been transformed to be in compliance with de-identification processes and checks. Please see supplementary files that describe the transformation and the alignment with statistical outputs reported in our 2025 Nature Medicine publication. The dataset is intended to support secondary analyses of clinical deterioration prediction and nursing surveillance-based decision support, and is released under the PhysioNet Restricted Health Data License 1.5.0. The Clinical Trial registration ID is NCT03911687.

Background

Hospitalized patients can deteriorate rapidly, and delayed recognition of early warning signs is a well-documented contributor to preventable in-hospital harm, including failure-to-rescue events, cardiac arrest, and death. Traditional early warning scores rely primarily on structured vital sign data and often miss the subtler, earlier signals of patient decline that experienced bedside nurses pick up through ongoing clinical surveillance — much of which is captured only in the frequency, timing, and pattern of nursing documentation rather than in discrete vital sign values.

The CONCERN (COmmunicating Narrative Concerns Entered by RNs) Early Warning System (EWS) was developed to close this gap. Rather than relying solely on physiological parameters, CONCERN’s machine learning algorithm models real-time nursing surveillance documentation patterns — for example, changes in the frequency or type of nursing assessments and interventions — as a proxy for clinical concern, and uses these patterns to generate a real-time deterioration risk score integrated into the electronic health record.

This database originates from a 1-year, multisite, pragmatic, cluster-randomized controlled trial (NCT03911687) designed to test whether informing care teams of CONCERN EWS risk scores improved patient outcomes compared with usual care [1]. The trial randomized 74 inpatient clinical units (37 intervention, 37 usual care) across two U.S. academic health systems, and enrolled 60,893 eligible adult hospital encounters (33,024 intervention, 27,869 usual care) over the study period.

Coprimary outcomes were in-hospital mortality (modeled as instantaneous risk) and hospital length of stay. Secondary outcomes included cardiopulmonary arrest, sepsis, unanticipated intensive care unit transfer, and 30-day hospital readmission. The trial found that encounters in units using the CONCERN EWS had significantly reduced risk of in-hospital death, shorter length of stay, and reduced risk of sepsis relative to usual care, alongside an increased rate of unanticipated ICU transfers — consistent with earlier identification of at-risk patients prompting escalation of care.

Widely used open clinical databases, such as MIMIC-IV (single center) [2] and the eICU Collaborative Research Database (multicenter, ICU-only) [3] are retrospective and observational. They support model development but cannot show whether a deployed early warning system improves outcomes. Prospective evidence is scarce: a systematic review found only 41 randomized trials of machine learning interventions in health care [4]. Most deterioration models also rely on physiologic and laboratory data [5], whereas nursing surveillance patterns carry earlier signal [6] and are rarely available openly. This database provides de-identified encounter-level outcomes from a pragmatic, cluster-randomized trial of the CONCERN early warning system across 74 units at two health systems [1], with known intervention assignment, enabling independent reanalysis and benchmarking. The dataset has been transformed from its original form to maintain compliance with release of a de-identified version. It is intended to support secondary analyses in clinical deterioration prediction, nursing-informatics-based decision support, and pragmatic trial methodology, and to allow independent verification and extension of the published findings.

Methods

Study context. This dataset derives from the CONCERN pragmatic, cluster-randomized controlled trial (NCT03911687). In the trial, 74 inpatient units (acute care units and ICUs) at two U.S. health systems (four hospitals) were randomized to display the CONCERN early warning system (EWS) risk score to the care team (37 units) or to usual care (37 units). Participants were enrolled between October 2020 and October 2022. Trial design, the intervention, and statistical methods are reported in full in Rossetti et al., Nature Medicine 2025 [1]. The remainder of this section describes only how the released dataset was created.

Source data. All data were extracted from the Epic electronic health record (EHR) at each site through SQL queries to the institutional data warehouse. The data were generated during routine clinical care rather than through study-specific instruments, so they reflect real-world documentation and coding practices.

Cohort. Each row is one hospital encounter that met the trial’s eligibility criteria: age 18 or older, hospitalized more than 24 hours, at least 12 hours on a study unit, and free of any in-hospital event (including discharge) for the first 6 hours on a study unit. Encounters with hospice or palliative care, or with DNR/DNI or comfort-care orders activated before an outcome event, were excluded. Of 79,049 encounters screened, 60,893 met these criteria and are included. Encounters were treated independently, and the identifier (ID) is a de-identified encounter identifier.

Exposure and setting variables. Patients are sometimes transferred between intervention and usual-care units, so the release provides summary indicators rather than unit-level histories. assignment_majority_int is 1 when the majority of a patient’s observed shift-level assignments were to intervention units. majority_icu is 1 when the majority of observed care-shift intervals were in an ICU. The CONCERN risk scores, nursing documentation metadata, and model inputs are not included.

Outcome variables. Outcomes were derived from the EHR using the trial definitions and are released as binary indicators:

• in-hospital mortality
• discharge alive
• sepsis (Sepsis-3: suspected infection plus SOFA or qSOFA criteria)
• cardiopulmonary arrest (Code Blue event)
• unanticipated ICU transfer (direct transfer from an acute care unit to the ICU, not resulting from a planned procedure)
• 30-day readmission (unplanned, at least 24 hours, same health system, among patients discharged alive)

Time to event was measured in 12-hour shifts. Length of stay was measured from inpatient admission to discharge or death and excludes pre-inpatient time such as emergency department stay.

Covariates. Age, sex, race, and ethnicity were taken from the EHR at admission. Race and ethnicity were recoded into the categories in the data dictionary. See [1] for methods to calculate Charlson Comorbidity Index. Study site is retained as a covariate.

De-identification. The release was derived from the internal encounter-level analytic dataset applying in a first phase HIPAA Safe Harbor followed by a second phase Expert Determination review which investigated and mitigated possible paths of re-identification. IRB approval was obtained from Columbia University with a waiver of consent. The following steps were applied:

• Direct identifiers, source-system identifiers, dates, timestamps, hospital-unit identifiers, and longitudinal clinical records were removed.
• Age was converted to categorical age groups.
• Charlson scores were grouped as 0, 1, 2, 3, 4, and 5 or more.
• Follow-up times for mortality, discharge, sepsis, cardiopulmonary arrest, and ICU transfer were coarsened into five intervals (1–2, 3–6, 7–14, 15–28, and 29 or more 12-hour shifts). The numeric values 2, 6, 14, 28, and 29 represent these intervals, not exact times.
• Length of stay was rounded to a 24-hour grid, perturbed with bounded random jitter (up to about 6 hours), and released in days.

Analytic validation. Accompanying R and SAS scripts reproduce the adjusted models from the released file. The reproduced estimates preserve the direction, magnitude, and statistical interpretation of the published results (Figure 1 of the documentation). Unit-level clustering cannot be reproduced because hospital unit was removed, so the scripts report model-based standard errors.

Data Description

The CONCERN de-identified dataset is an encounter-level analytic dataset created to support reproduction of models reported in the CONCERN study at the shift level. Each row represents one eligible hospital encounter and shift. The released CSV contains demographic characteristics, study assignment, clinical-setting indicators, binary study outcomes, coarsened follow-up variables, and a transformed length-of-stay measure. It does not contain direct identifiers, dates, timestamps, hospital-unit identifiers, or detailed longitudinal clinical records.

See list of files and description for each provided here:

1. CONCERN_Deidentified_Dataset_File_and_Variable_Description.txt: Overview, file description, variable dictionary, de-identification methods and analytic utility validation forest plot, basic data summary, usage notes and limitations
2. concern_deidentified_release_20260724.csv: De-identified encounter & shift-level analytic dataset
3. concern_reproduce_results_20260724.R: R code that reproduces the validation models and forest plot
4. concern_reproduce_results_20260724.sas: SAS code that reproduces the validation models
5. concern_reproduced_forest_20260724.png: Forest plot comparing published estimates to the reproduced estimates
6. concern_reproduction_sessionInfo_20260724.txt: R session information at the time of the reproduction

Usage Notes

The dataset is intended to support secondary analyses of clinical deterioration prediction and nursing surveillance-based decision support. This release is intended to reproduce the selected adjusted CONCERN models. It is not a complete electronic health record extract and should not be used to reconstruct individual clinical trajectories or exact event timing.

Coarsened follow-up values do not preserve exact timing within a bin. The value 29 represents 29 or more shifts and should not be interpreted as exactly 29 shifts.

The released length-of-stay measure differs intentionally from the original duration because of rounding and jitter.

Hospital unit is not included. The original unit-level clustered standard errors therefore cannot be reproduced from the released dataset. The accompanying reproduction script reports model-based standard errors and does not substitute site-level clustering.

Analyses requiring exact timestamps, unit transitions, repeated measurements, or additional clinical covariates cannot be conducted with this release.

Release Notes

Version 1.0.0

Initial release

Ethics

This study was approved by Columbia University IRB.

Clinical trial # NCT03911687

Informed Consent was waived by the IRB.

Acknowledgements

This study was funded by the National Institute of Nursing Research (NINR R01NR016941, COmmunicating Narrative Concerns Entered by RNs (CONCERN): Clinical Decision Support Communication for Risky Patient States) and and Reducing Health Disparities Through Informatics (T32NR007969). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. We thank all the nurses, prescribing providers and patients who participated in this study. We also thank Amy Finnegan for her independent validation of the database de-identification.

Conflicts of Interest

Columbia University has filed a US nonprovisional patent application (US Patent Application No. 18/814,823) related to the technology that was evaluated using this database. Sarah Rossetti, Chris Knaplund and Kenrick Cato are named inventors on the patent application and are entitled to revenue sharing with the university per the terms of the university’s patent policy. The university and the named inventors are committed to making the technology freely available upon request for academic noncommercial research purposes. Any entity interested in obtaining a license to practice the technology for commercial purposes may contact Columbia University Technology Ventures.

References

1. Rossetti SC, Dykes PC, Knaplund C, Cho S, Withall J, Lowenthal G, Albers D, Lee RY, Jia H, Bakken S, Kang MJ, Chang FY, Zhou L, Bates DW, Daramola T, Liu F, Schwartz-Dillard J, Tran M, Bokhari SMA, Thate J, Cato KD. Real-time surveillance system for patient deterioration: a pragmatic cluster-randomized controlled trial. Nat Med. 2025 Jun;31(6):1895-1902. doi:10.1038/s41591-025-03609-7.
2. Johnson AEW, Bulgarelli L, Shen L, et al. MIMIC-IV, a freely accessible electronic health record dataset. Sci Data. 2023;10:1. doi:10.1038/s41597-022-01899-x
3. Pollard TJ, Johnson AEW, Raffa JD, Celi LA, Mark RG, Badawi O. The eICU Collaborative Research Database, a freely available multi-center database for critical care research. Sci Data. 2018;5:180178. doi:10.1038/sdata.2018.178
4. Plana D, et al. Randomized clinical trials of machine learning interventions in health care: a systematic review. JAMA Netw Open. 2022;5:e2233946.
5. Fu L-H, et al. Development and validation of early warning score system: a systematic literature review. J Biomed Inform. 2020;105:103410.
6. Rossetti SC, et al. Healthcare process modeling to phenotype clinician behaviors for exploiting the signal gain of clinical expertise (HPM-ExpertSignals): development and evaluation of a conceptual framework. J Am Med Inform Assoc. 2021;28:1242–1251.