EHRNote-ChatQA: A Benchmark for Evidence-Grounded Multi-Turn Clinical Question Answering over Longitudinal Discharge Summaries

@article{PhysioNet-ehrnote-chatqa-1.0.0,
  author = {Kim, Jiyoun and Yeo, Muhan and Jang, Eunhye and Yang, Jeewon and Yoon, Hangyul and Lee, Su Ji and Han, Heejo and Jung, Heejae and Kwon, Doyun and Lee, Jun Young and Lee, Jaehun and Lee, Jung Oh and Kweon, Sunjun and Moon, Jong Hak and Kim, Daseul and Cho, Minjae and Choi, Edward},
  title = {{EHRNote-ChatQA: A Benchmark for Evidence-Grounded Multi-Turn Clinical Question Answering over Longitudinal Discharge Summaries}},
  journal = {{PhysioNet}},
  year = {2026},
  month = oct,
  note = {Version 1.0.0},
  doi = {10.13026/x533-f455},
  url = {https://doi.org/10.13026/x533-f455}
}

@article{pollard_physionet_2026, title = {{PhysioNet} as a global platform for biomedical research}, volume = {1}, issn = {3005-0693}, url = {https://doi.org/10.1038/s44360-026-00096-z}, doi = {10.1038/s44360-026-00096-z}, abstract = {Twenty-five years after its first public repository was established, the PhysioNet resource for biomedical and clinical data has enabled tens of thousands of research papers. Further efforts are ongoing to advance secure data sharing, community engagement and scalable research infrastructure.}, number = {8}, journal = {Nature Health}, author = {Pollard, Tom and Moody, Benjamin E. and Lehman, Li-wei H. and Gow, Brian J. and Fernandes, Chrystinne and Xie, Chen and Johnson, Alistair and Mark, Roger G. and Heldt, Thomas}, month = aug, year = {2026}, pages = {792--795},}

Abstract

EHRNote-ChatQA is a benchmark for evidence-grounded, multi-turn clinical question answering over longitudinal discharge summaries. Derived from de-identified MIMIC-IV data, it contains 967 patient-level samples covering one to five discharge summaries per patient and 16,072 medical-expert-verified multiple-choice questions.

The dataset includes 8,036 clinical content questions and 8,036 paired evidence-grounding questions across eight categories: diagnosis, symptom, procedure, medication, microbiology, clinical assessment, clinical outcome, and discharge plan. Each content question is followed by a question asking which evidence-grounding source supports the preceding answer. All questions are five-option multiple choice.

The questions within each sample are designed to reflect realistic inquiry patterns used by medical experts when reviewing a patient’s discharge summaries. The dataset was constructed through an expert-informed pipeline combining a medical-expert-designed discharge-summary structuring schema, expert-curated multi-turn question-answering templates, and large language model-based data generation. Every patient-level question-answering sample was subsequently reviewed and, when necessary, revised by a medical expert to ensure clinical realism, accuracy, and reliable evidence grounding.

Background

Clinical review of discharge summaries is often longitudinal and iterative. Medical experts may need to determine what occurred during one admission, how a condition was managed, whether it recurred, and what changed during later admissions. This process is challenging because discharge summaries may be written by different clinicians, follow inconsistent formats, contain extensive medical abbreviations, and accumulate across admissions. As a result, answering a single clinical question may require reviewing and synthesizing information from several long, fragmented documents.

Two capabilities are particularly important for automated systems in this setting. The first is maintaining accuracy across a sequence of related questions, because clinical inquiry often proceeds through follow-up questions that build on prior context rather than through isolated queries [1]. The second is grounding each answer in exact, traceable evidence from the patient record. Language models may produce fluent but hallucinated or unverifiable responses [2], making it important for medical experts to be able to verify each answer against specific portions of the source documents [3].

Most existing clinical question-answering benchmarks do not evaluate these capabilities jointly. MedQA [4], MedMCQA [5], and PubMedQA [6] evaluate medical knowledge or literature comprehension without using a specific patient record as the basis for the questions. Benchmarks built from patient notes, including emrQA [7], EHRNoteQA [8], and DiSCQ [9], are limited to single-turn question answering and do not pair each content question with a question asking for its supporting source. Among resources that include evidence annotations, emrQA [7] and ArchEHR-QA [10] operate within a single clinical document and do not evaluate evidence grounding across a sequence of related questions spanning a patient’s longitudinal discharge summaries.

EHRNote-ChatQA was created to support reproducible evaluation of these capabilities in a longitudinal, patient-specific setting.

Methods

EHRNote-ChatQA was constructed from de-identified discharge summaries in MIMIC-IV [11]. All note text originates from MIMIC-IV and retains the de-identification applied in the source dataset. An initial cohort of 1,000 patients was sampled across five note-count groups, with 200 patients having each of one, two, three, four, or five discharge summaries. Patients were allocated uniformly across eight clinical question categories, yielding 25 patients for each category–note-count combination.

The construction process was developed in collaboration with four medical experts: three clinicians and one nurse. The experts defined a discharge-summary structuring schema consisting of 19 clinical entity types, 3–10 attributes per entity type, and 23 relationship types. Four example entity types and their attributes are shown below.

Relationships connect these entities to represent clinical information in the discharge summaries, such as Medication → administered_for → Diagnosis, Medication → improves → Symptom, Lab_Test → reveals → Finding, and Diagnosis → resulted_in → Outcome.

The experts also designed 80 multi-turn question-answering templates across eight clinical categories, with five templates per category for each of the single-note and multi-note settings. The templates reflected realistic patterns of clinical record review, including cross-admission inquiries concerning disease course, recurrence, and changes in discharge plans. They served as exemplars for LLM-based data generation rather than as rule-based templates for slot filling. An example single-note template from the Diagnosis category is shown below.

Gemini 2.5 Pro [12] was used to generate patient-specific, multi-turn question sequences from each patient’s complete set of discharge summaries. The model was provided with the discharge-summary structuring schema and the expert-curated question-answering templates as guidance. Each content question was generated in a five-option multiple-choice format containing one correct answer and four incorrect answer choices.

The incorrect choices included both note-grounded distractors, based on incorrect interpretations of information in the patient’s records, and clinical-knowledge-grounded distractors, consisting of clinically plausible content not documented in the records. This design requires models both to interpret the discharge summaries accurately and to apply relevant clinical knowledge when distinguishing the correct answer from plausible alternatives.

Each content question was paired with an evidence-grounding question that asks which source locations—identified by note number, chart date, and section header or headers—support the preceding answer. The correct evidence-grounding choice was required to be minimal yet sufficient: all clinical details in the preceding answer had to be supported by the selected sections, while no redundant section could be included when its supporting information was already contained in another selected section.

All 1,000 initially generated patient-level samples were reviewed by one of 11 medical experts over a three-week period. The review team consisted of nine specialist clinicians, one clinician resident, and one nurse. Reviewers assessed the clinical naturalness of the multi-turn question sequence, the correctness of the questions and correct answer choices, the validity of the incorrect answer choices, and the sufficiency and minimality of the evidence-grounding choices. Each sample was reviewed by a single assigned medical expert; no subset underwent independent review by multiple medical experts. This design reflected the dataset’s scale (16,072 QA pairs) and limited expert availability.

During medical expert review, 33 samples were removed because their overall question sequences could not be revised into clinically realistic inquiries, or because incorrectly de-identified section headers in the source notes made evidence-grounding questions unanswerable. Other flagged samples contained errors in individual questions or answer choices that could be corrected while preserving a valid, source-grounded sequence. The final dataset contains 967 patient-level samples. Among the retained content questions, 125 questions (1.6% of 8,036), 101 correct answer choices (1.3%), and 156 incorrect answer choices (0.5% of 32,144) were revised. For the evidence-grounding questions, 425 correct answer choices (5.3% of 8,036) and 1,676 incorrect answer choices (5.2% of 32,144) were revised. The positions of the correct answers were subsequently shuffled to produce an approximately uniform distribution across choices A through E.

All use of MIMIC-IV-derived content with Gemini 2.5 Pro during data generation was conducted through a HIPAA-compliant Google Cloud Vertex AI deployment, consistent with the MIMIC-IV data use agreement.

Data Description

The dataset is provided as two CSV files, notes.csv and questions.csv. It contains 967 patient-level, multi-turn question-answering samples, with one sample corresponding to each patient.

The dataset includes:

• 967 patients;
• 2,852 discharge summaries;
• one to five discharge summaries per patient;
• 8,036 clinical content questions;
• 8,036 evidence-grounding questions; and
• 16,072 questions in total.

The distribution of samples by the number of discharge summaries is as follows:

The dataset covers eight primary question categories:

notes.csv

The columns in notes.csv are described below:

questions.csv

The columns in questions.csv are described below:

Usage Notes

EHRNote-ChatQA is intended for evaluating clinical language models, retrieval-augmented systems, agentic question-answering pipelines over longitudinal patient records, and more. It supports analysis of multi-turn clinical question answering and evidence identification with source attribution.

The dataset has some limitations. First, the benchmark is constructed from data from a single hospital system (MIMIC-IV), and therefore may not fully capture variation across institutions or EHR systems. Second, we focus on discharge summaries, which provide the overall clinical context of each admission, but do not cover other EHR sources such as progress notes, nursing notes, radiology reports, or structured EHR tables. Extending evidence-grounded multi-turn QA to these heterogeneous data sources may be an important direction for future work. Finally, although every QA sample is reviewed and revised by medical experts, each sample is reviewed by a single expert rather than multiple independent annotators. This trade-off was necessary given the large scale of the dataset (16,072 QA pairs) and the limited availability of medical experts.

For more details of EHRNote-ChatQA, please reference our official paper.

Release Notes

Version 1.0.0

Initial release

Ethics

This dataset is derived from MIMIC-IV and exists under the same IRB. No new patient data are introduced in this dataset.

Conflicts of Interest

The authors have no conflicts of interest to declare.

References

1. Ely JW, Osheroff JA, Chambliss ML, Ebell MH, Rosenbaum ME. Answering physicians' clinical questions: obstacles and potential solutions. J Am Med Inform Assoc. 2005;12(2):217–224.
2. Ji Z, Lee N, Frieske R, et al. Survey of hallucination in natural language generation. ACM Comput Surv. 2023;55(12):1–38.
3. Bohnet B, Tran VQ, Verga P, et al. Attributed question answering: Evaluation and modeling for attributed large language models. arXiv:2212.08037. 2022.
4. Jin D, Pan E, Oufattole N, Weng WH, Fang H, Szolovits P. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. Applied Sciences. 2021;11(14):6421.
5. Pal A, Umapathi LK, Sankarasubbu M. MedMCQA: A large-scale multi-subject multi-choice dataset for medical domain question answering. In: Proceedings of the Conference on Health, Inference, and Learning (CHIL). 2022:248–260.
6. Jin Q, Dhingra B, Liu Z, Cohen WW, Lu X. PubMedQA: A dataset for biomedical research question answering. In: Proceedings of EMNLP-IJCNLP. 2019:2567–2577.
7. Pampari A, Raghavan P, Liang J, Peng J. emrQA: A large corpus for question answering on electronic medical records. In: Proceedings of EMNLP. 2018:2357–2368.
8. Kweon S, Kim J, Kwak H, et al. EHRNoteQA: An LLM benchmark for real-world clinical practice using discharge summaries. In: Advances in Neural Information Processing Systems. 2024.
9. Lehman E, Lialin V, Legaspi KE, et al. Learning to ask like a physician. In: Proceedings of the 4th Clinical Natural Language Processing Workshop. 2022:74–86.
10. Soni S, Demner-Fushman D. Overview of the ArchEHR-QA 2025 shared task on grounded question answering from electronic health records. In: Proceedings of the BioNLP Workshop. 2025.
11. Johnson AEW, Bulgarelli L, Shen L, et al. MIMIC-IV, a freely accessible electronic health record dataset. Sci Data. 2023;10:1.
12. Comanici G, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv:2507.06261. 2025.
13. Kim J, Yeo M, Jang E, Yang J, Yoon H, Lee SJ, Han HJ, Jung HJ, Kwon D, Lee J, Lee JO. EHRNote-ChatQA: A Benchmark for Evidence-Grounded Multi-Turn Clinical Question Answering over Longitudinal Discharge Summaries. arXiv preprint arXiv:2606.15735. 2026 Jun 14.