EHR-ReasonCon: Reasoning-Intensive Consistency Verification Between Clinical Notes and Structured Tables in Electronic Health Records

@article{PhysioNet-ehr-reasoncon-1.0.0,
  author = {Kwon, Yeonsu and Kim, Jiho and Choi, JunSeong and Rabaey, Paloma and Kim, Minseo and Im, Sujeong and Yang, Jeewon and JUNMIN, LEE and Lee, Sangji and Kim, Jiwon and Yoon, Hangyul and Kwon, Hyunwook and Choi, Edward},
  title = {{EHR-ReasonCon: Reasoning-Intensive Consistency Verification Between Clinical Notes and Structured Tables in Electronic Health Records}},
  journal = {{PhysioNet}},
  year = {2026},
  month = sep,
  note = {Version 1.0.0},
  doi = {10.13026/7m91-3010},
  url = {https://doi.org/10.13026/7m91-3010}
}

@article{pollard_physionet_2026, title = {{PhysioNet} as a global platform for biomedical research}, volume = {1}, issn = {3005-0693}, url = {https://doi.org/10.1038/s44360-026-00096-z}, doi = {10.1038/s44360-026-00096-z}, abstract = {Twenty-five years after its first public repository was established, the PhysioNet resource for biomedical and clinical data has enabled tens of thousands of research papers. Further efforts are ongoing to advance secure data sharing, community engagement and scalable research infrastructure.}, number = {8}, journal = {Nature Health}, author = {Pollard, Tom and Moody, Benjamin E. and Lehman, Li-wei H. and Gow, Brian J. and Fernandes, Chrystinne and Xie, Chen and Johnson, Alistair and Mark, Roger G. and Heldt, Thomas}, month = aug, year = {2026}, pages = {792--795},}

Abstract

Electronic Health Records (EHRs) combine structured data, such as medications and laboratory measurements, with unstructured clinical notes describing patient conditions and clinical events. Although these two data types provide complementary information, discrepancies may arise between them and potentially affect patient safety and clinical decision-making. Existing datasets for note-table consistency verification mainly focus on direct matching of numerical values or individual events. However, real-world clinical documentation often requires medical interpretation, reasoning over relations between events, and analysis of changes over time. To address this limitation, we developed EHR-ReasonCon, a reasoning-intensive dataset for verifying consistency between clinical notes and structured EHR tables. EHR-ReasonCon was constructed from MIMIC-III in collaboration with clinical experts and contains manual annotations of 8,042 entities from 105 clinical notes. Each entity was checked against records from 14 structured tables and labeled as CONSISTENT or INCONSISTENT.

Background

Electronic Health Records are digital collections of patient information generated during clinical care. EHRs generally include both structured tables, such as laboratory results, medications, diagnoses, and vital signs, and unstructured clinical notes, such as physician and nursing notes. These two forms of data are interconnected throughout clinical workflows. Practitioners review information recorded in structured tables, interpret the patient’s condition, and document diagnoses, treatments, and clinical progress in narrative notes. As a result, reliable agreement between clinical notes and structured records is important for understanding a patient’s clinical course. However, inconsistencies may arise because EHR systems are often designed around administrative requirements rather than clinical documentation workflows [1]. Practitioners may also enter inaccurate values, copy outdated information, omit relevant records, or document events at incorrect times [2, 3]. These discrepancies may reduce the reliability of EHR data and create risks for patient care and secondary data analysis [4]. Manual verification of clinical notes against large relational databases is time-consuming and expensive. EHRCon introduced a benchmark for automatically checking consistency between clinical notes and structured EHR tables [5]. However, EHRCon primarily focuses on surface-level information, such as whether a numerical value or a simple clinical event is present in the database. Clinical notes frequently describe information that cannot be verified through direct matching alone. For example, confirming anemia requires interpreting hemoglobin values, verifying antibiotic sensitivity requires connecting multiple microbiology fields, and validating a decreasing blood pressure trend requires examining measurements across several time points. EHR-ReasonCon was designed to capture these reasoning-intensive consistency verification cases.

Methods

EHR-ReasonCon was constructed on MIMIC-III [6] through a five-stage, expert-guided annotation pipeline, using the 105 clinical notes included in EHRCon [5] as the source notes. Eight trained annotators familiar with EHR systems performed the annotation, consulting authoritative medical references (UpToDate, MedlinePlus, Cleveland Clinic, Mayo Clinic) and physicians as needed.

Stage 0 — Pre-Annotation Setup. In collaboration with four clinical experts (a board-certified radiation oncologist, a board-certified general surgeon, an anesthesiology resident, and an EHR technician), we developed an annotation protocol specifying how narrative expressions in notes map to structured table fields, how to interpret temporal trends, and how to handle ambiguous clinical judgments. The protocol was iteratively refined through pilot annotation studies. During piloting, we analyzed how annotators searched for evidence and formalized the recurring search patterns into eight modular table-exploration tools, organized into three functional categories: (1) entity-to-table-item alignment (lexical and semantic retrieval of table items corresponding to a note entity, e.g., aligning "WBC" with "White Blood Cells"); (2) database exploration and value profiling (schema exploration and summaries of typical values for each item, e.g., recognizing that Stool Amount is categorical rather than numerical); and (3) temporal and conditional record retrieval (retrieval of records by time window and value constraints to inspect trends and events).

Stage 1 — Anchor Entity Identification. Annotators identified anchor entities in each note that correspond to items in structured tables (e.g., medications, diagnoses, vitals, labs). These anchors serve as entry points for evidence retrieval.

Stage 2 — Tool-Assisted Table Exploration. For each anchor entity, annotators reviewed its attributes (e.g., dosage, route) and temporal context and used the predefined tools to query the structured tables and retrieve the records that serve as evidence for consistency assessment.

Stage 3 — Consistency Verification. Each anchor entity was labeled CONSISTENT if the corresponding note information is supported by the structured records, and INCONSISTENT otherwise. This stage involves reasoning-intensive analysis — temporal reasoning, commonsense reasoning, and medical interpretation — and consultation of established medical references or physicians when necessary.

Stage 4 — Annotation Reliability. For each note, two annotators independently completed Stages 1–3, then resolved disagreements through mutual reconciliation, escalating complex clinical judgments to medical professionals. After the initial annotations were complete, an independent reviewer re-examined all 105 notes to ensure dataset-wide consistency. The resulting inter-annotator agreement was 0.897 for entity identification and 0.888 for consistency labeling.

Data Description

The dataset contains 8,042 annotated anchor entities drawn from 105 clinical notes across three note types, each linked to structured records in 14 MIMIC-III tables (Chartevents, Labevents, Prescriptions, Inputevents_cv, Inputevents_mv, Outputevents, Procedureevents_mv, Microbiologyevents, Diagnoses_icd, Procedures_icd, D_items, D_icd_diagnoses, D_icd_procedures, and D_labitems). The data are split into test and validation sets at a 4:1 ratio.

1. Folder Hierarchy

The folder structure is shown below:

EHR_ReasonCon/
├── README.md                                   # file guide (formats, file relationships)
├── annotations.csv                             # all 8,042 annotations in one flat table
├── discharge/
│   ├── test/
│   │   ├── discharge_test.csv                  # source notes for this split
│   │   └── EHRReasonCon_{hadm_id}_data.json    # one annotation file per note
│   └── valid/
│       ├── discharge_val.csv
│       └── EHRReasonCon_{hadm_id}_data.json
├── physician/
│   ├── test/  ...
│   └── valid/ ...
├── nursing/
│   ├── test/  ...
│   └── valid/ ...
├── patient_data/
│   └── {hadm_id}/                              # per-admission structured tables
│       ├── chartevents.csv
│       ├── labevents.csv
│       ├── prescriptions.csv
│       ├── inputevents_cv.csv
│       ├── inputevents_mv.csv
│       ├── outputevents.csv
│       ├── procedureevents_mv.csv
│       ├── microbiologyevents.csv
│       ├── diagnosis_icd.csv
│       ├── procedure_icd.csv
│       ├── patient_records.csv                 # item catalog recorded for this patient
│       └── catalog_items.csv                   # candidate items with profiles (TOP_VALUES)
└── resources/
    ├── preprocessed_d_items.csv                # normalized D_ITEMS dictionary used by tools
    ├── d_labitems.csv                          # D_LABITEMS dictionary for labevents.ITEMID
    └── top_values.json                         # corpus-wide value profiles per item

• Each note's source text is provided as a CSV ({notetype}_{split}.csv) containing the standard MIMIC-III NOTEEVENTS columns (ROW_ID, SUBJECT_ID, HADM_ID, CHARTDATE, CHARTTIME, CATEGORY, TEXT, ADMITTIME, etc.). The patient_data/{hadm_id}/ folders provide the structured tables restricted to that admission, allowing every entity to be verified against real records without loading the full MIMIC-III database.

2. Annotation file format

Each EHRReasonCon_{hadm_id}_data.json is a JSON object keyed by the note's ROW_ID. The value is a list of annotated entities; each entity is a single-key object mapping the entity name to its annotation metadata. Files load with the standard json library. annotations.csv contains the same 8,042 annotations flattened to one row per entity (keyed by hadm_id, row_id, entity_index) for users who prefer a single table; the per-note JSON files are authoritative. README.md documents how notes, annotations, and structured tables reference one another.

• data - The value(s) extracted from the note, mapped to the corresponding table item(s) and column(s) (e.g., {'labevents': {'valuenum': '1.4'}, 'd_labitems': {'label': 'creatinine'}}).
• label - Consistency label for the entity: consistent or inconsistent.
• entity_type - Verification type inherited from the annotation schema: 1 = numerical value, 2 = existence-verifiable event, 3 = string/categorical value.
• position - Line number of the entity within the note.
• table - The primary structured table used as evidence (e.g., chartevents).
• evidence_row_ids - List of ROW_IDs of the table records that serve as supporting evidence.
• consistency_check_path - Coded trace of the tool-exploration/reasoning path taken by the annotator to reach the label.
• inconsistent_columns - For inconsistent entities, the specific column(s) where the discrepancy occurred (e.g., charttime, valuenum).
• almost_consistent - Whether the note and table values are near-identical with a negligible difference (Yes/No).
• commonsense_medical_none - Flag indicating whether commonsense (c), medical (m), or no (n) domain knowledge was required.
• medical_knowledge_source - URL of the authoritative medical reference consulted, when applicable.
• check_db - Whether database or guideline consultation was required to reach the label.
• abnormal_case - Flag marking an atypical or edge case noted during annotation.

Not every field is present for every entity; optional fields are populated only when relevant to that entity's verification.

Usage Notes

EHR-ReasonCon is intended for developing and evaluating systems that verify consistency between clinical notes and structured EHR tables, with an emphasis on reasoning-intensive cases involving clinical interpretation, event relations, and temporal change. Because span boundaries and granularity naturally differ between systems and human annotators, we recommend semantic, entity-level evaluation (e.g., LLM-as-a-judge under harsh and lenient criteria, as used in the accompanying paper) rather than exact string matching.

To load an annotation file:

import json 
import pandas as pd 

notes = pd.read_csv("discharge/test/discharge_test.csv") 
note = notes.iloc[0] # HADM_ID = 115468.0, ROW_ID = 1087 

with open(f"discharge/test/EHRReasonCon_{note.HADM_ID}_data.json", encoding="utf-8") as f: 
   annotations = json.load(f) # {"1087": [ {entity: {...}}, ... ]} 

entities = annotations[str(note.ROW_ID)] 
records = pd.read_csv(f"patient_data/{int(note.HADM_ID)}/{table}.csv") 
# or, as a single flat table flat = pd.read_csv("annotations.csv")

The accompanying paper introduces EHR-INSPECTOR, an LLM-based framework that segments notes, extracts anchor entities and temporal references, and verifies consistency using table-exploration tools; EHR-ReasonCon serves as its evaluation benchmark. Limitations include the reliance on preprocessed MIMIC-III data, the focus on adult ICU records inherent to MIMIC-III, and the annotation scope of 105 notes; broader coverage would require additional annotation with the same protocol.

Release Notes

1.0.0 - Initial Release

Ethics

This project builds upon previous datasets (MIMIC-III) that have been approved for credentialed distribution.

Conflicts of Interest

The authors have no conflicts of interest to declare.

References

1. L. B. Villa and I. Cabezas, “A review on usability features for designing electronic health records,” in Proc. 2014 IEEE 16th International Conference on e-Health Networking, Applications and Services (Healthcom), 2014, pp. 49–54.
2. T. H. Payne, W. D. Alonso, J. A. Markiel, K. Lybarger, R. Lordon, M. Yetisgen, J. M. Zech, and A. A. White, “Using voice to create inpatient progress notes: Effects on note timeliness, quality, and physician satisfaction,” JAMIA Open, vol. 1, no. 2, pp. 218–226, 2018.
3. A. Y. Tsou, C. U. Lehmann, J. Michel, R. Solomon, L. Possanza, and T. Gandhi, “Safe practices for copy and paste in the EHR: Systematic review, recommendations, and novel model for health IT collaboration,” Applied Clinical Informatics, vol. 8, no. 1, pp. 12–34, 2017.
4. A. W. Demsash, S. Y. Kassie, et al., “Health professionals’ routine practice documentation and its associated factors in a resource-limited setting: A cross-sectional study,” BMJ Health & Care Informatics, vol. 30, no. 1, Art. no. e100699, 2023.
5. Y. Kwon, J. Kim, G. Lee, S. Bae, D. Kyung, W. Cha, T. Pollard, A. Johnson, and E. Choi, “EHRCon: Dataset for checking consistency between unstructured notes and structured tables in electronic health records,” in Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2024.
6. A. Johnson, T. Pollard, and R. Mark, MIMIC-III Clinical Database, version 1.4. PhysioNet, Sep. 2016.
7. G. R. Cohen, C. P. Friedman, A. M. Ryan, C. R. Richardson, and J. Adler-Milstein, “Variation in physicians’ electronic health record documentation and potential patient harm from that variation,” Journal of General Internal Medicine, vol. 34, no. 11, pp. 2355–2367, 2019.
8. S. T. Rosenbloom, J. C. Denny, H. Xu, N. Lorenzi, W. W. Stead, and K. B. Johnson, “Data from clinical notes: A perspective on the tension between structure and flexible documentation,” Journal of the American Medical Informatics Association, vol. 18, no. 2, pp. 181–186, 2011.
9. D. Newman-Griffis, G. Divita, B. Desmet, A. Zirikly, C. P. Rosé, and E. Fosler-Lussier, “Ambiguity in medical concept normalization: An analysis of types and coverage in electronic health record datasets,” Journal of the American Medical Informatics Association, vol. 28, no. 3, pp. 516–532, 2021.
10. Y. Kwon, J. Kim, J. Choi, P. Rabaey, M. Kim, S. Im, J. Yang, J.-M. Lee, S. Lee, J. Kim, H. Yoon, H. Kwon, and E. Choi, “Towards error-free EHRs: Reasoning-intensive consistency verification between clinical notes and structured tables in electronic health records,” *arXiv preprint arXiv:2605.26463*, 2026.