ER-REASON: A Benchmark Dataset for LLM-Based Clinical Reasoning in the Emergency Room

@article{PhysioNet-er-reason-1.1.0,
  author = {Molina, Mel and Mehandru, Nikita and Golchini, Niloufar and Alaa, Ahmed},
  title = {{ER-REASON: A Benchmark Dataset for LLM-Based Clinical Reasoning in the Emergency Room}},
  journal = {{PhysioNet}},
  year = {2026},
  month = nov,
  note = {Version 1.1.0},
  doi = {10.13026/c0zm-0m96},
  url = {https://doi.org/10.13026/c0zm-0m96}
}

@article{pollard_physionet_2026, title = {{PhysioNet} as a global platform for biomedical research}, volume = {1}, issn = {3005-0693}, url = {https://doi.org/10.1038/s44360-026-00096-z}, doi = {10.1038/s44360-026-00096-z}, abstract = {Twenty-five years after its first public repository was established, the PhysioNet resource for biomedical and clinical data has enabled tens of thousands of research papers. Further efforts are ongoing to advance secure data sharing, community engagement and scalable research infrastructure.}, number = {8}, journal = {Nature Health}, author = {Pollard, Tom and Moody, Benjamin E. and Lehman, Li-wei H. and Gow, Brian J. and Fernandes, Chrystinne and Xie, Chen and Johnson, Alistair and Mark, Roger G. and Heldt, Thomas}, month = aug, year = {2026}, pages = {792--795},}

Abstract

The ER-Reason dataset is a benchmark designed to evaluate LLM-based clinical reasoning and decision-making in the emergency room (ER), a high-stakes setting where clinicians make rapid, consequential decisions across diverse patient presentations and medical specialties under time pressure. This longitudinal collection of de-identified clinical notes encompasses 3,437 patients admitted to the ER at a large academic medical center between March 1, 2022, and March 31, 2024. ER-Reason contains 25,174 notes spanning discharge summaries, progress notes, history and physical exams, consults, echocardiography reports, imaging notes, and ER provider documentation across 3,984 encounters. The benchmark includes evaluation tasks from key stages of the ER workflow: triage intake, initial assessment, treatment selection, disposition planning, and final diagnosis, each structured to reflect core clinical reasoning processes such as differential diagnosis via rule-out reasoning. We also collected 72 full physician-authored rationales explaining reasoning processes that mimic the teaching process used in residency training, and are typically absent from ER documentation. This retrospective dataset captures unstructured, multi-encounter clinical notes reflecting the real-world complexity of ER patient care.

Background

As more advanced LLMs such as Deepseek-R1 [1], OpenAI o1 [2], and o3-mini [3] that are designed for complex reasoning emerge, there is a need for new benchmark datasets that enable rigorous evaluation of the ability of LLMs to emulate clinician reasoning in real-world clinical settings.

We introduce ER-Reason, a benchmark designed to evaluate LLM-based clinical reasoning and decision-making in the emergency room (ER). The ER provides a unique setting for evaluating LLMs for multiple reasons. First, emergency physicians must make rapid, high-stakes decisions based on fragmented, heterogeneous clinical documentation, a setting where LLMs could support clinicians by synthesizing and contextualizing disparate sources of information. Second, ER decision-making follows a structured workflow that involves multiple interdependent stages—triage, assessment, treatment, and disposition—each influenced by both patient-specific factors and systemic constraints such as hospital capacity and acuity levels [4]. Third, unlike many other clinical settings, ER provider notes often contain explicit justifications for diagnostic and treatment choices. Due to the episodic nature of emergency care, clinicians adopt a "worst-first" approach that prioritizes the exclusion of life-threatening conditions over comprehensive care, and must clearly communicate their reasoning to downstream providers [5, 6]. The ER provider note thus serves as a communication tool that reflects clinical reasoning, including which differential diagnoses were considered, which were ruled out, and the rationale behind key decisions. However, due to the intense pace and workload of the ER, these notes can also under-document the full scope of clinician reasoning [7].

ER-Reason captures the full trajectory of patients in the ER at a large academic medical center, with data reflecting the unique characteristics of ER care described above. The benchmark includes:

• LLM evaluation tasks aligned with key stages of the ER workflow, including triage intake, EHR review, initial assessment, treatment selection, disposition planning, and final diagnosis. Each task is grounded in realistic input derived from the patients’ longitudinal clinical documentation.

• Annotations capturing clinical reasoning behind decision-making at each stage of the ER workflow. In addition to clinical rationale documented in ER provider notes, the benchmark includes 72 expert-authored rationales curated from practicing ER residents and physicians, with structured annotations of reasoning that mimic the teaching processes found in residency training.

• To support evaluation of sequential clinical reasoning, two board-certified emergency physicians created Script Concordance Test (SCT) annotations, a validated framework for assessing belief updating under certainty [8], from 194 patient encounters derived from the ER Provider notes in ER-Reason.

Methods

Patient and Encounter Selection:
The dataset includes all adult patients (ages 18–85) who were admitted to the Emergency Room (ER) at UCSF between March 1, 2022, and March 31, 2024. Only encounters with a specified primary ED diagnosis were included (i.e., encounters with PrimaryEdDiagnosisName not equal to *Unspecified).

Note Selection and Filtering:
For each included encounter, the most recent note of each type created before the encounter’s discharge date was included in the dataset. The note types included were:

• ED Provider Notes
• Discharge Summaries
• Progress Notes
• History & Physical (H&P) Notes
• Consults
• Imaging Reports
• Echocardiography (ECG) Reports

Social work notes and other note types were excluded to provide additional privacy protection. On average, there are seven notes per patient encounter.

Encounter-Level Inclusion Criteria:
Encounters were included in the final dataset only if at least one Discharge Summary and one ED Provider Note were present. Some patients may have multiple encounters within the study period.

Demographics and Clinical Data:
For each patient, demographic information includes sex, race, age at encounter (calculated from birthdate and encounter departure date), preferred language, highest level of education, and marital status. Chief complaint and admission/discharge information (year only) are also included, with dates shifted to preserve privacy.

Expert Rationales:
To support clinical reasoning evaluation, the dataset includes 72 expert-authored rationales from emergency medicine residents and attending physicians. Participants followed a structured workflow in a custom application to document step-by-step reasoning, including rule-out logic, relevant medical factors, and treatment planning. IRB approval was obtained, and participants were compensated.

Benchmark Tasks:
The dataset supports five benchmark tasks aligned with ER workflow stages:

1. Triage Intake: Assess patient acuity based on initial clinical notes.
2. Patient Case Summarization: Generate a one-line summary of the patient’s case using prior discharge notes and the current chief complaint.
3. Treatment Planning: Predict differential diagnoses, identify relevant medical factors (labs, imaging, clinical findings), and propose treatment plans.
4. Final Diagnosis: Assign definitive diagnoses using ICD-10 codes and HCC categories based on all available clinical information.
5. Disposition Decision: Recommend patient disposition (e.g., discharge, admit, transfer) informed by clinical presentation and history.

Evaluation Metrics:
Each task specifies inputs, outputs, and evaluation metrics. Metrics include ROUGE/F1 for summaries, ICD-10 and Hierarchical Condition Category (HCC) accuracy for diagnoses, and clinical concept recall using Unified Medical Language System (UMLS) Concept Unique Identifiers (CUIs). No preprocessing beyond de-identification was applied before release.

Measuring Clinical Reasoning via SCTs:

To support evaluation of sequential clinical reasoning, two board-certified emergency physicians created Script Concordance Test (SCT) reasoning annotations from 194 patient encounters derived from the chief complaint, ER provider note, and the final diagnosis. These 194 patient encounters were randomly sampled from the larger ER-Reason cohort. For each case, two attending physicians independently annotated each timestep, with a third serving as adjudicator for disagreements. At each timestep, annotators assigned an ordinal belief-update score (-2 to +2) to each of the five differentials, indicating how the new evidence changed the likelihood of that diagnosis. Annotators also ranked the full differential after each timestep and provided a final ranked differential with a one-sentence rationale at the end of each case. Each annotation is linked to the base dataset via encounterkey.

Evaluation Metrics:

Each SCT case yields two types of physician-assigned scores. DxUpdate is an ordinal belief-update score (−2 to +2) assigned to each differential at each timestep, where −2 indicates the new evidence strongly decreases the likelihood of that diagnosis, 0 indicates no change, and +2 indicates the new evidence strongly increases the likelihood of the diagnosis. DxTrajectory is a ranked list of the five differentials assigned by the physician after each timestep, reflecting the updated differential ranking as new evidence accumulates. The DxTrajectory at the final timestep serves as the physician's concluding ranked differential, or final diagnosis. Agreement between model and physician scores can be quantified using linearly weighted Cohen's $\kappa$ for DxUpdate and Spearman's $\rho$ for DxTrajectory.

De-Identification:
All notes were de-identified by the University of California, San Francisco (UCSF) Information Commons using both HIPAA Safe Harbor and HIPAA Expert Determination methods. Safe Harbor standards were applied to remove or mask standard identifiers, while the Expert Determination method was additionally employed to allow for date-shifting.

Data Description

Unlike standardized note formats such as SOAP (subjective, objective, assessment, and plan), the variety of note types and unstructured formats in ER-REASON present additional complexity for clinical reasoning tasks. The dataset includes 395 unique chief complaints among admitted patients, with the most common being abdominal pain, shortness of breath, and chest pain. For each admitted ER patient, we include clinical notes from their previous hospital encounters, capturing the complexity and temporal progression of patient care across multiple visits. To our knowledge, ER-REASON is the first publicly released dataset providing longitudinal, multi-encounter clinical notes for ER patients, enabling evaluation of LLMs on realistic clinical reasoning tasks.

Each row in er_reason.csv represents a single patient encounter. Multiple notes from the same encounter are linked via unique identifiers, and historical notes (discharge summaries, progress notes, H&P, imaging, ECGs, consults, echocardiograms) are included alongside the current visit’s ED provider note. Expert-authored rationales are captured in the Rule_Out, Decision_Factors, and Treatment_Plan columns, providing step-by-step clinical reasoning related to the chief complaint, demographics, and one-line summary (One_Sentence_Extracted). Chief complaints are standardized and stored in primarychiefcomplaintname.

Special Notes on Year Fields
Some date-related fields (e.g., ArrivalYearKey, DepartureYearKey) may contain the value 1970. This value does not indicate an actual event year, but instead reflects a default or placeholder (commonly a Unix epoch fallback). Users should interpret 1970 as indicating missing or unavailable date information.

For the SCT reasoning evaluation we have two csvs: sct_test and sct_reasoning. The first contains the raw test cases derived from the 194 patient encounters. The second consists of the ground-truth physician annotations after adjudication. A breakdown of the columns in each csv can be found below:

sct_test.csv

sct_reasoning.csv

icd10_codes.csv

* structured as diagnosis-code rows
* note that there are only 3,964 unique encounters compared to the 3,984 unique encounters in the full cohort. This is due to the fact that the remaining 20 encounters did not have a diagnosis code assigned at the time of the ED encounter.

Usage Notes

The ER-REASON dataset can be used for research purposes such as:

• Benchmarking LLMs on ER note summarization and multi-encounter reasoning.
• Studying clinical decision-making workflows in the emergency setting.
• Training models to predict diagnoses, treatment plans, and patient disposition from sequential clinical notes.

Limitations:

• The dataset is intended for research use only and not for clinical decision-making.
• Users must comply with the dataset license and maintain data privacy standards.

While the notes are de-identified, they should not be used in any context that could attempt to re-identify patients.

For code, examples, and discussion of the dataset, see the associated GitHub repository [9].

Release Notes

Version 1.1.0

This is the stable release, and the schema and structure are not expected to change. ER-Reason v1.1.0 adds the physician annotations (sct_reasoning.csv, sct_test.csv) and structured icd10 codes (icd10_codes.csv).

Version 1.0.0

Initial release

Ethics

The collection and rigorous de-identification of patient information were conducted by the Information Commons team at the University of California, San Francisco (UCSF). Approval to share this dataset was granted by the institution’s compliance team.

Acknowledgements

We would like to thank the University of California, San Francisco (UCSF) Information Commons for their continued support of the ER-Reason project. In particular, we are grateful to Albert Lee for his invaluable assistance, as well as to Helena Mezgova and Ariel Deardorff for their guidance and oversight on compliance matters.

Conflicts of Interest

None to declare.

References

1. Guo D, Yang D, Zhang H, Song J, Zhang R, Xu R, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv [Preprint]. 2025 Jan 22. doi:10.48550/arXiv.2501.12948.
2. Zhong T, Liu Z, Pan Y, Zhang Y, Zhou Y, Liang S, et al. Evaluation of OpenAI o1: Opportunities and Challenges of AGI. arXiv [Preprint]. 2025 Jul 7. doi:10.48550/arXiv.2409.18486.
3. Mondillo G, Masino M, Colosimo S, Perrotta A, Frattolillo V. Evaluating AI reasoning models in pediatric medicine: A comparative analysis of O3-mini and O3-mini-high. medRxiv [Preprint]. 2025 Feb 27. doi:10.1101/2025.02.27.25323028
4. Kanzaria HK, Brook RH, Probst MA, Harris D, Berry SH, Hoffman JR. Emergency physician perceptions of shared decision-making. Acad Emerg Med. 2015;22(4):399–405.
5. Croskerry P. A universal model of diagnostic reasoning. Acad Med. 2009;84(8):1022–8.
6. Kellermann AL, Hsia RY, Yeh C, Morganti KG. Emergency care: then, now, and next. Health Aff (Millwood). 2013;32(12):2069–74.
7. Hill RG Jr, Sears LM, Melanson SW. 4000 clicks: a productivity analysis of electronic medical records in a community hospital ED. Am J Emerg Med. 2013;31(11):1591–4.
8. Charlin B, Roy L, Brailovsky C, Goulet F, van der Vleuten C. The script concordance test: a tool to assess the reflective clinician. Teach Learn Med. 2000;12(4):189–195.
9. AlaaLab. ER-Reason [Internet]. GitHub; 2025. Available from: https://github.com/AlaaLab/ER-Reason/ [Accessed 29 Sept 2025].

Cited by

4 publications cite this project. The most recent are listed below.

• Deriving a Deployment Threshold for Benefit and Harm in Clinical AI: Multi-Institutional Decision-Analytic Modeling Study
  Yngve Mikkelsen. JMIR AI, 2026.
• Multi-agent LLM diagnostic teams improve differential recall on curated cases and reverse on real emergency notes: a label-level reversal without a clinician-judged safety difference
  Jun Feng, Yuhao Jiao, Huaiya Xie et al. medRxiv, 2026.
• Quantify, Collect, and Correct: Evidence-Driven Multi-agent Framework for Critical Evidence Refinement in Emergency Rooms
  Haoyu Cao, Yueze Fu, Wenbo Gao et al. Lecture notes in computer science, 2026.
• Cross-Site Stability of Embedding Model Rankings for Known-Item Retrieval From Clinical Notes: A Two-Corpus Comparative Evaluation Study (Preprint)
  Yngve Mikkelsen. JMIR AI, 2026.

View all 4 publications

Citation data from OpenAlex. Papers that use the project without citing it may be missing.