MIMIC-IV-Ext-DeVisE. Counterfactual ICU Admission Notes for Behavioral Evaluation of Medical Language Models

@article{PhysioNet-mimic-iv-ext-devise-1.0.0,
  author = {{Zurdo Tagliabue}, Camila and Calixto, Iacer and Erdem, Erkut and {Oss Boll}, Heloisa and Erdem, Aykut},
  title = {{MIMIC-IV-Ext-DeVisE. Counterfactual ICU Admission Notes for Behavioral Evaluation of Medical Language Models}},
  journal = {{PhysioNet}},
  year = {2026},
  month = oct,
  note = {Version 1.0.0},
  doi = {10.13026/r89h-ed36},
  url = {https://doi.org/10.13026/r89h-ed36}
}

@article{pollard_physionet_2026, title = {{PhysioNet} as a global platform for biomedical research}, volume = {1}, issn = {3005-0693}, url = {https://doi.org/10.1038/s44360-026-00096-z}, doi = {10.1038/s44360-026-00096-z}, abstract = {Twenty-five years after its first public repository was established, the PhysioNet resource for biomedical and clinical data has enabled tens of thousands of research papers. Further efforts are ongoing to advance secure data sharing, community engagement and scalable research infrastructure.}, number = {8}, journal = {Nature Health}, author = {Pollard, Tom and Moody, Benjamin E. and Lehman, Li-wei H. and Gow, Brian J. and Fernandes, Chrystinne and Xie, Chen and Johnson, Alistair and Mark, Roger G. and Heldt, Thomas}, month = aug, year = {2026}, pages = {792--795},}

Abstract

MIMIC-IV-Ext-DeVisE contains counterfactual ICU admission notes designed to enable controlled behavioral evaluation of medical large language models (LLMs) to clinically meaningful perturbations in patient data. It was created as part of the DeVisE (Demographics and Vital Signs Evaluation) framework.

The dataset includes 1,001 ICU admission notes derived from MIMIC-IV and 135,861 single-variable counterfactual perturbations: 25,025 for demographics (age, sex, ethnicity) and 110,836 for vital signs (heart rate, blood pressure -systolic and diastolic- , respiratory rate, oxygen saturation, and temperature). Vital-sign perturbations are additionally provided in both unstructured and template-based note formats, yielding 246,697 counterfactual records in total across all JSONL files. All other clinical content is preserved, enabling controlled analysis of model sensitivity to clinically meaningful changes.

Two complementary formats are provided: (i) unstructured clinical text derived from discharge summaries and (ii) template-based structured notes containing only variables of interest also derived from discharge summaries. Counterfactual values are sampled from clinically grounded severity ranges to ensure realism and interpretability.

The dataset supports research on robustness, fairness, and clinical reasoning in medical LLMs by enabling controlled, minimally differing input comparisons.

Background

LLMs are increasingly used for clinical decision support and medical question answering, demonstrating strong performance across a range of tasks [1,2]. However, standard medical benchmarks primarily evaluate predictive accuracy and offer limited insight into whether models apply clinical knowledge in a manner consistent with established medical reasoning, or whether their predictions reflect spurious correlations and shortcuts.

A key question in clinical NLP evaluation is whether models respond appropriately to changes in patient state. For instance, if a patient's heart rate increases from a normal value (89 bpm) to a high value (120 bpm), a well-calibrated model should both assign lower probability to the counterfactual patient profile (since the new profile is less typical) and predict worse clinical outcomes, such as higher mortality risk or longer ICU length-of-stay. Standard accuracy metrics do not capture this kind of behavioral consistency [3].

The DeVisE framework addresses this gap through behavioral testing with minimally differing counterfactuals, a paradigm originally developed in software engineering [4] and subsequently adapted for NLP [5] and vision-and-language tasks [6]. By isolating the effect of single-variable changes, DeVisE enables controlled probing of model sensitivity and directional reasoning across both demographic and physiological variables. MIMIC-IV-Ext-DeVisE was created to support such evaluations in DeVisE: Towards the Behavioral Testing of Medical Large Language Models [7], and to serve as a reusable, clinically grounded benchmark for the broader research community.

Methods

Data Source

The dataset is derived from MIMIC-IV version 3.1, a large, publicly available database of de-identified ICU patient records [8]. MIMIC-IV includes both structured data (demographics, vital signs, laboratory results, medications) and free-text discharge notes. The benchmark includes 1,001 admission notes, randomly selected from MIMIC-IV after excluding patients missing all vital signs (retaining only patients with no more than four missing vital signs).

From Discharge to Admission Notes

Because the downstream tasks evaluated in DeVisE, mortality and ICU length-of-stay (LoS) prediction, can use only information available at admission, discharge summaries were preprocessed to remove all information pertaining to the ICU stay itself. Following Van Aken et al. [3], the following fields were retained: chief complaint, present illness, medical history, admission medications, allergies, physical examination, family history, and social history.

Clinical Variables

Demographics: age (binned into four classes: young adults 18–35, middle-aged adults 36–55, older adults 56–75, and elderly 76–91), gender (female/male), and ethnicity (Asian & Pacific, Black, Hispanic/Latino, Other/Unknown, White). Demographic variables were extracted directly from MIMIC-IV structured data.

Vital signs: heart rate (HR), systolic blood pressure (SBP), diastolic blood pressure (DBP), respiratory rate (RR), oxygen saturation (O2Sat), and body temperature (T). Vital signs were extracted from the physical examination section of admission notes with LLM few-shot prompting.

Vital sign extraction used LLaMA-3.3-70B-Instruct (Meta AI, 70B parameters) [13], an open-weight model, with synthetic few-shot prompting applied to the physical examination section of each admission note. The model was run on the Snellius national supercomputer (SURF, the Netherlands), so no third-party API or external data retention was involved. The model was instructed to extract temperature, heart rate, blood pressure, respiration rate, and oxygen saturation, and to return them as a single JSON object, guided by two worked examples; the full prompt template is provided in the accompanying paper’s appendix [7].

Severity Scales

Numerical vital sign variables were mapped onto clinically grounded severity bins, very low (-2), low (-1), normal (0), high (1), and very high (2), derived from clinical guidelines [9–11]. These severity categories provide a consistent, interpretable scale for counterfactual analysis and enable aggregation of results across vital signs. Oxygen saturation is a special case: values above the normal range are not clinically defined, so only three severity levels apply (very low, low, normal).

Counterfactual Generation

Counterfactuals were generated by modifying one variable at a time while keeping all other content fixed. For each patient and each variable, five counterfactual values were sampled uniformly from each severity class, yielding balanced coverage across all classes. For categorical demographic variables, counterfactuals cover all alternative class values. Counterfactual text modifications were applied programmatically to the original notes.

The generation pipeline consisted of five steps:

1. Extracting demographic variables from MIMIC-IV structured data.

2. Extracting vital sign values from admission notes using an LLM with few-shot prompting.

3. Generating counterfactual values according to clinically grounded severity scales.

4. Automatically verifying that edits affected only the intended variable, using the GNU diff tool.

5. Manual validation to identify and correct extraction errors.
   Manual validation was carried out by reviewing all extracted and replacement values across the raw and template-based notes (original and counterfactual) against the source admission note text, identifying and correcting extraction errors [7]. This review found an error rate of approximately 5% in vital signs extracted via the LLM, and no errors in demographic replacements drawn directly from structured MIMIC-IV data; all identified errors were manually corrected

For more details, see the DeVisE code repository [12].

Template-Based Notes

In addition to raw admission notes, a template-based version of each note was created containing only the variables of interest in a structured format (age, gender, ethnicity, and the six vital signs). Template-based notes allow for the evaluation of model behavior in the absence of confounding contextual information, isolating responses to specific variable changes.

Data Description

The dataset is distributed as a collection of JSONL files with the following structure:

• original_notes_unstructured.jsonl — Raw admission notes derived from MIMIC-IV discharge summaries.

• original_notes_template_based.jsonl — Template-based versions of the same notes.

• demographics_counterfactuals_template_based.jsonl — Counterfactual examples for demographic variables (age, gender, ethnicity) in template-based format.

• vital_signs_counterfactuals_template_based.jsonl — Counterfactual examples for all six vital signs in template-based format.

• vital_signs_counterfactuals_unstructured.jsonl — Counterfactual examples for all six vital signs in raw (unstructured) note format.

Each JSONL entry for unstructured notes contains: subject_id, hadm_id, text (the full admission note), los_icu_hours, and mortality_label. Template-based entries include structured fields for age, sex, race, and individual vital sign values. Counterfactual entries include additional metadata: a unique id (formatted as hadm-id_variable_severity-score_n-counterfactual), variable_modified, original_severity, counterfactual_severity, original_severity_score (on a scale of -2 to +2), and counterfactual_severity_score.

Data Dictionary

Field definitions for each file are provided in the data dictionary below. Note that outcome labels (los_icu_hours, mortality_label) are only present in the two original (non-counterfactual) files; counterfactual records can be joined back to these via hadm_id when outcome labels are needed.

original_notes_unstructured.jsonl

1,001 records - one per admission note

original_notes_template_based.jsonl

1,001 records - structured version of the same notes

demographics_counterfactuals_template_based.jsonl

25,025 records - single-variable demographic counterfactuals

vital_signs_counterfactuals_template_based.jsonl

110,836 records - single-variable vital-sign counterfactuals, template-based format

vital_signs_counterfactuals_unstructured.jsonl

110,836 records - single-variable vital-sign counterfactuals, unstructured note format

Breakdown of Number of Counterfactual Variables

The counts below are unique single-variable counterfactual perturbations (systolic and diastolic blood pressure are counted once under “Blood pressure”, since they are modified together). They sum to 135,861 unique perturbations: 25,025 demographic + 110,836 vital-sign. Demographic perturbations are provided only as template-based notes (25,025 JSONL records). Vital-sign perturbations are additionally provided as unstructured notes, so each of the 110,836 vital-sign perturbations appears in two files, yielding 221,672 records; combined with the demographic records, the dataset totals 246,697 counterfactual JSONL records across all files.

• Age: 20,020

• Sex: 1,001

• Ethnicity: 4,004

• Blood pressure (systolic & diastolic, varied jointly): 30,895

• Heart Rate: 27,095

• Temperature: 20,590

• Respiratory Rate: 19,404

• Oxygen Saturation: 12,852

Usage Notes

This dataset is intended for researchers evaluating LLM behavior in clinical settings. Primary use cases include:

• Behavioral testing of medical LLMs using controlled counterfactual perturbations.

• Robustness analysis: assessing how stable model predictions are under minimally differing inputs.

• Fairness and bias analysis: examining whether demographic variables (age, gender, ethnicity) systematically influence model predictions in ways that reflect or amplify known healthcare disparities.

• Sensitivity analysis: measuring whether models respond proportionally and directionally correctly to changes in physiological severity.

The dataset was used in the evaluation reported in DeVisE: Towards the Behavioral Testing of Medical Large Language Models [7], which benchmarked eight LLMs, including both general-purpose and medical-domain models, across mortality and ICU length-of-stay prediction tasks. Key findings include that template-based notes substantially amplify model sensitivity to vital sign changes compared to raw notes, and that demographic perturbations produce small but systematic prediction shifts consistent with known healthcare disparities.

Known limitations include: (1) data are drawn from a single US hospital and may not generalise to other healthcare systems or languages; (2) only demographic and vital sign variables are covered, other clinically relevant variables are not included; (3) counterfactual values are sampled uniformly within severity bins, which may yield values that are rare in practice.

Code for dataset generation and model evaluation is available at DeVisE repository [12]. The repository includes preprocessing scripts, counterfactual generation code, experiment configurations, and evaluation metrics.

Users must comply with the MIMIC-IV data usage agreement and the PhysioNet credentialed access policy before using this dataset.

Release Notes

Version 1.0.0

Initial release

Ethics

This dataset is derived from MIMIC-IV, which contains de-identified clinical data collected under institutional review board approval. The use of the data complies with HIPAA regulations and PhysioNet credentialing requirements.

The primary benefit of this dataset is to support rigorous, clinically grounded evaluation of medical LLMs, contributing to safer deployment in real-world healthcare settings. A potential risk is that findings relating to demographic variables could be misinterpreted; users are encouraged to interpret results in the context of the broader literature on healthcare disparities and model fairness.

Acknowledgements

H.O.B. and I.C. are funded by the project CaRe-NLP (file number NGF.1607.22.014) of the research programme AiNed Fellowship Grants, financed in part by the Dutch Research Council (NWO). E.E. is funded by the TUBITAK 2247-A National Outstanding Researchers Program (Award No. 123C542). We thank the MIMIC-IV contributors and the PhysioNet team.

Conflicts of Interest

The author(s) have no conflicts of interest to declare.

References

1. Singhal K, Tu T, Gottweis J, Sayres R, Wulczyn E, Amin M, Hou L, Clark K, Pfohl SR, Cole-Lewis H, Neal D. Toward expert-level medical question answering with large language models. Nature medicine. 2025 Mar;31(3):943-50.
2. Van Veen D, Van Uden C, Blankemeier L, Delbrouck JB, Aali A, Bluethgen C, Pareek A, Polacin M, Reis EP, Seehofnerová A, Rohatgi N. Adapted large language models can outperform medical experts in clinical text summarization. Nature medicine. 2024 Apr;30(4):1134-42.
3. Van Aken B, Papaioannou JM, Mayrdorfer M, Budde K, Gers F, Loeser A. Clinical outcome prediction from admission notes using self-supervised knowledge integration. InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume 2021 Apr (pp. 881-893).
4. Beizer B, Wiley J. Black box testing: Techniques for functional testing of software and systems. iEEE Software. 1996 Sep 30;13(5):98.
5. Ribeiro MT, Wu T, Guestrin C, Singh S. Beyond accuracy: Behavioral testing of NLP models with CheckList. InProceedings of the 58th annual meeting of the association for computational linguistics 2020 Jul (pp. 4902-4912).
6. Kesen I, Pedrotti A, Dogan M, Cafagna M, Acikgoz EC, Parcalabescu L, Calixto I, Frank A, Gatt A, Erdem A, Erdem E. ViLMA: A zero-shot benchmark for linguistic and temporal grounding in video-language models. In: Proceedings of ICLR; 2024.
7. Zurdo Tagliabue C, Oss Boll H, Erdem A, Erdem E, Calixto I. DeVisE: Towards the Behavioral Testing of Medical Large Language Models. In: Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL). 2026.
8. Johnson AE, Bulgarelli L, Shen L, Gayles A, Shammout A, Horng S, Pollard TJ, Hao S, Moody B, Gow B, Lehman LW. MIMIC-IV, a freely accessible electronic health record dataset. Scientific data. 2023 Jan 3;10(1):1.
9. American Heart Association. Understanding blood pressure readings and heart rate guidelines. AHA. 2024.
10. Royal College of Physicians. National Early Warning Score (NEWS) 2. RCP. 2017.
11. Freifeld AG, Baden LR, Brown AE, Elting LS, Gelfand M, Greene JN, Ito JI, King E, Marcucci G, Montoya JG, Morris A. Fever and neutropenia clinical practice guidelines. Journal of the National Comprehensive Cancer Network: JNCCN. 2004 Sep 1;2(5):390-432.
12. DeVisE repository available at: https://github.com/camztag/DeVisE
13. 13. Grattafiori A, Dubey A, Jauhri A, Pandey A, Kadian A, Al-Dahle A, Letman A, Mathur A, Schelten A, Vaughan A, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. 2024.