ReEvalMed: Rethinking Medical Report Evaluation by Aligning Metrics with Real-World Clinical Judgment

@article{PhysioNet-reevalmed-1.0.0,
  author = {Li, Ruochen and Xia, Yufei and Zhu, Youxiang and Li, Jun and Jian, Bailiang and Yuan, Kun},
  title = {{ReEvalMed: Rethinking Medical Report Evaluation by Aligning Metrics with Real-World Clinical Judgment}},
  journal = {{PhysioNet}},
  year = {2026},
  month = oct,
  note = {Version 1.0.0},
  doi = {10.13026/esap-z328},
  url = {https://doi.org/10.13026/esap-z328}
}

@article{pollard_physionet_2026, title = {{PhysioNet} as a global platform for biomedical research}, volume = {1}, issn = {3005-0693}, url = {https://doi.org/10.1038/s44360-026-00096-z}, doi = {10.1038/s44360-026-00096-z}, abstract = {Twenty-five years after its first public repository was established, the PhysioNet resource for biomedical and clinical data has enabled tens of thousands of research papers. Further efforts are ongoing to advance secure data sharing, community engagement and scalable research infrastructure.}, number = {8}, journal = {Nature Health}, author = {Pollard, Tom and Moody, Benjamin E. and Lehman, Li-wei H. and Gow, Brian J. and Fernandes, Chrystinne and Xie, Chen and Johnson, Alistair and Mark, Roger G. and Heldt, Thomas}, month = aug, year = {2026}, pages = {792--795},}

Abstract

Automatically generated radiology reports often receive high scores from existing evaluation metrics but fail to earn clinicians' trust. This gap reveals fundamental flaws in how current metrics assess the quality of generated reports. We rethink the design and evaluation of these metrics and propose a clinically grounded Meta-Evaluation framework. We define clinically grounded criteria spanning clinical alignment and key metric capabilities, including discrimination, robustness, and monotonicity. Using a fine-grained dataset of ground truth and rewritten report pairs annotated with error types, clinical significance labels, and explanations, we systematically evaluate existing metrics and reveal their limitations in interpreting clinical semantics, such as failing to distinguish clinically significant errors, over-penalizing harmless variations, and lacking consistency across error severity levels. Our framework offers guidance for building more clinically reliable evaluation methods. The full dataset contains 420 report pairs in total, consisting of 400 main aspect-level pairs and 20 calibration pairs used for monotonicity analysis.

Background

Automatic radiology report generation has advanced rapidly with the development of vision–language models [1,2]. However, widely used automatic evaluation metrics often fail to reflect clinical reliability [3]. Reports with high metric scores may still contain clinically significant errors that can mislead clinical decision-making and reduce clinician trust [3,4,5].

Most widely used metrics rely on surface-level textual similarity, such as BLEU [6] and ROUGE-L [7], or on structured matching against predefined labels or entities, such as CheXbert-based classification [8] and RadGraph [9]. These approaches primarily capture lexical or entity overlap and lack a deeper understanding of the clinical meaning of a report. More recent large language model–based metrics, such as GREEN [10] and GEMA-Score [3], partially address this limitation, but they typically inherit a coarse scoring scheme built on the six error categories of the ReXVal dataset [11,12] and therefore omit several clinically relevant dimensions (for example, size and distance, uncertainty expression, internal contradiction, and descriptive accuracy). Moreover, the clinical validity of such metrics is commonly assessed by correlating their scores with aggregate radiologist error counts from ReXVal; because inter-annotator agreement on those counts is limited, a high correlation alone is insufficient to establish that a metric is clinically reliable [12,13]. These limitations motivate clinically grounded evaluation resources that test whether metrics respond appropriately to clinically meaningful, rather than merely textual, differences.

What distinguishes this dataset from prior resources. Existing resources for radiology-report evaluation, most notably ReXVal dataset, primarily evaluate naturally occurring errors in model-generated reports. They collect naturally occurring errors in model-generated reports and provide aggregate, per-category error counts without recording the clinical reasoning behind each judgment, and they validate metrics by correlation with those counts. In this work, we use GT to denote the ground-truth reference report and ME to denote a meta-evaluation rewrite, namely a controlled modified version of the GT report. ReEvalMed is designed as a controlled diagnostic stress-test dataset. Each GT–ME pair injects a single, known error along three explicitly controlled factors — clinical aspect, error type, and clinical significance — and is accompanied by an expert explanation. This factorized, explanation-bearing design moves evaluation from "agreement with human counts" to capability-level probing: it allows a metric's behavior to be attributed to a specific clinical dimension and supports the direct measurement of discrimination, robustness, and monotonicity. To our knowledge, this is the first controlled radiology-report evaluation resource built specifically to test report-evaluation metrics under controlled, single-aspect perturbations with documented clinical rationale.

Methods

The dataset was constructed to support clinically grounded meta-evaluation of automatic radiology report evaluation metrics. We focused on chest X-ray reports and used only the findings section as the ground-truth content.

Source selection and sampling. Ground-truth passages were drawn from two publicly available, credentialed datasets: MIMIC-CXR [14,15,16] and ReXVal [11,12]. From MIMIC-CXR we randomly sampled 50 chest X-ray reports and retained only their findings sections. From ReXVal we additionally selected 20 information-rich reports — reports whose findings contained multiple, clinically substantive observations — as identified by a clinician, because sparse or near-empty findings provide little material into which a controlled error can be introduced. Only the findings text was used; impressions, demographic fields, and image data were not used. No patient-level or demographic attributes were incorporated. Eligible MIMIC-CXR reports were chest X-ray reports with a non-empty findings section containing at least one clinically substantive observation.

Exclusions. No additional source reports were excluded after source selection. During dataset construction, rewrite candidates that failed validation were revised, regenerated, or excluded before inclusion in the released dataset.

Criteria development and clinical guidance. We first defined a clinical aspect taxonomy and an aspect-specific criteria table to distinguish clinically significant from clinically insignificant modifications. The criteria were developed by the authors through clinical consultation and were informed by established reporting and management guidance, including Fleischner Society recommendations [17], ACR Lung-RADS [18], and SCCT [19] guidance where applicable. The final taxonomy includes anatomical location, disease severity, lesion description, negation, modality appropriateness, size and distance, temporal comparison/progression, uncertainty expression, internal contradiction, terminology usage, linguistic noise, and stylistic variation.

Rewriting procedure. For each selected ground-truth report, we produced one or more meta-evaluation rewrites (ME) by introducing a single controlled modification targeting one clinical aspect at a time. Rewrites were generated with a locally deployed DeepSeek-R1 model using aspect-specific prompts derived from the criteria table. Each rewrite modifies only one aspect while preserving the surrounding clinical context, so that the introduced difference is isolated and interpretable. Each rewrite instantiates one of three error types: omission (O; removal of clinically relevant information), fabrication (F; introduction of unsupported clinical information), or inaccuracy (E; inconsistent description of the same clinical finding). The label S does not denote a fourth error type; instead, it indicates a pooled setting for a given clinical aspect, where individual rewrites still contain a single error type but omission, fabrication, and inaccuracy are sampled in approximately balanced proportions rather than separated into distinct aspect-error cells.

Although the prompts were adapted to the target aspect, error type, and significance level, they followed a common structure. A representative prompt template is shown below. The locally deployed model was used only to generate candidate rewrites; all generated candidates were subsequently reviewed, corrected, selected, or excluded through the clinical validation workflow described below.

Representative prompt template:
You are assisting with the construction of a radiology report evaluation dataset. Given the ground-truth findings section of a chest X-ray report, generate one controlled candidate rewrite by modifying exactly one clinical aspect. The rewrite should match the requested clinical-significance level: a clinically significant change is one that could plausibly affect clinical interpretation, follow-up, treatment, urgency, or downstream decision-making; a clinically insignificant change preserves the same clinical meaning and would not plausibly affect downstream decisions.
Ground-truth findings:{GT_FINDINGS}
Target clinical aspect:{CLINICAL_ASPECT}
Target error type:{ERROR_TYPE}
Target clinical significance:{SIGNIFICANCE_LEVEL}
Error type definitions:
Omission:A clinically relevant fact present in the GT is missing in the ME. 
Fabrication: The ME introduces a clinically relevant fact that is not present in the GT.
Inaccuracy: The same clinical fact is described inconsistently between GT and ME.
Instructions:
1. Modify only the specified clinical aspect.
2. Introduce only one controlled change. Instantiate only the specified error type.
3. Match the requested clinical-significance level.
4. Preserve unrelated clinical content and the overall reporting style where possible.
5. Do not introduce patient identifiers, demographic information, or any new protected health information.
6. Avoid multiple interacting changes that would make the clinical-significance judgment ambiguous.
Output:
ME: <rewritten findings>
Explanation: <brief rationale for why the modification is clinically significant or clinically insignificant>

Assignment of clinical significance. For each clinical aspect, we constructed variants at both clinically significant and clinically insignificant levels using aspect-specific prompts derived from the criteria table. For example of Size/Distance aspect, changing a 3-cm mass to an 8-cm mass was treated as clinically significant, whereas changing 3 cm to 2.9 cm was treated as clinically insignificant. To prevent metrics from relying only on superficial textual similarity, significant pairs retained substantial shared context so that the injected error was the primary deviation, whereas insignificant pairs were trimmed to the modified region where appropriate. The generated rewrites were subsequently validated through the clinical review workflow described below.

Annotation and validation workflow. All GT-ME pairs were reviewed and validated by a physician co-author with 4-5 years of clinical experience and experience in interpreting radiology reports. The review was conducted against the same aspect-specific criteria table used during rewrite generation, which was informed by established clinical guidance, including Fleischner Society recommendations [17], ACR Lung-RADS [18], and SCCT [19] guidance where applicable. For each pair, the reviewer verified that the rewrite isolated the intended clinical aspect and error type, confirmed or corrected the clinical-significance label, selected clinically plausible cases for inclusion, and wrote a free-text explanation documenting why the modification would or would not plausibly affect downstream clinical interpretation or decision-making. Pairs that introduced multiple interacting changes, failed to isolate the targeted aspect, or had a significance label that could not be justified under the criteria were revised, regenerated, or excluded. We intentionally modified only one clinical aspect per pair to keep the evaluation setting controlled and to avoid ambiguity about which change caused the clinical-significance judgment. Because each pair was reviewed by one clinician, there was no independent multi-rater adjudication. When the clinician's judgment differed from the initial prompt-assigned label, the clinician's judgment was treated as final. Ambiguous cases were discussed with the author team and were revised, regenerated, or excluded rather than retained with an uncertain label.

The final released dataset contains 420 GT–ME pairs generated from these 70 source reports. Of these, 400 pairs were used for discriminative and robustness evaluation across 12 clinical aspects, and 20 pairs formed a separate monotonicity calibration subset. For the four aspects — location, severity, description, and comparison/progression — we generated 10 significant and 10 insignificant pairs for each of the three error types, yielding 4 × 3 × 2 × 10 = 240 pairs. For the remaining eight aspects, we generated 10 significant and 10 insignificant pairs per aspect, with error types sampled in approximately balanced proportions, yielding 8 × 2 × 10 = 160 pairs. Together, these form the 400 aspect-level pairs.

Data Description

The dataset is released as a single nested JSON file, reevalmed_dataset.json.

The top level of the JSON file is a list of 20 objects, one for each discrete significance value. Each top-level object contains two fields: significance, an integer from 0 to 19, and data, an array of GT–ME report pairs:

    [
      { "significance": 0,  "data": [ {pair}, {pair}, ... ] },
      { "significance": 1,  "data": [ ... ] },
      ...
      { "significance": 19, "data": [ ... ] }
    ]

Each significance value contains 21 GT–ME pairs. The full dataset therefore contains 420 report pairs in total, consisting of 400 main aspect-level pairs and 20 calibration pairs used for monotonicity analysis.

The significance value is stored at the top level and applies to all pair objects in the corresponding data array. Values 0–9 denote clinically insignificant variations, while values 10–19 denote clinically significant errors. The fine-grained 0–19 scale is used to preserve the ordered structure needed for the calibration subset to do monotonicity analysis. For the discrimination and robustness analyses, the operative distinction is the binary split between insignificant values 0–9 and significant values 10–19.

The clinical aspect label is represented by the type field. Its possible values are: Location, Severity, Description, Comparison/Progression, Negation, Modality, Size/distance, Contradiction, Noise, Stylistic Variation, Terminology, Uncertainty, and Calibration. Calibration is not a clinical aspect; it marks a pair as belonging to the monotonicity subset. There is exactly one Calibration pair per significance value (20 in total), and every pair has error_type = S.

The error_type field uses the following values:

• O denotes omission, where clinically relevant information present in the GT is removed from the ME.
• F denotes fabrication, where the ME introduces unsupported clinical information that is not present in the GT.
• E denotes inaccuracy, where the same clinical finding is described inconsistently between the GT and ME.
• S denotes a mixed-error setting. It is not a fourth error type. Instead, it indicates that omission, fabrication, and inaccuracy examples are sampled in approximately balanced proportions within that clinical aspect or calibration subset.

Each discrete significance value contains 21 GT–ME pairs: 12 pairs from the four fully crossed aspects (Location, Severity, Description, and Comparison/Progression), where omission, fabrication, and inaccuracy are represented separately; 8 mixed-error pairs from the remaining clinical aspects, recorded as error_type = S; and 1 calibration pair, also recorded as error_type = S. Thus, each significance value contains 3 × 4 + 8 + 1 = 21 pairs. Across the ten clinically significant values (10–19), each represented aspect–error cell contains 10 pairs, one for each value; the same structure is mirrored across the ten clinically insignificant values (0–9), matching the per-column structure of Tables 4 and 5 in the accompanying publication [13].

Files in the release.

• reevalmed_dataset.json: the main dataset file described above. This is the only released data file.
• README.md: an overview of the dataset, file list, schema, and getting-started instructions.
• run_metric_demo.ipynb: a minimal reference notebook that loads the dataset and reproduces the BLEU column of the Discriminative/Robustness tables and the BLEU monotonicity trajectory; provided for illustration and not required to use the data.

All reports are derived from de-identified source reports. No new protected health information or identifiable information was introduced during rewriting or annotation.

Usage Notes

This dataset is intended for the evaluation and meta-evaluation of automatic radiology report evaluation metrics, not for training or fine-tuning report-generation models. A typical workflow applies a candidate metric to each (GT, ME) pair, treating GT as the reference and ME as the comparison, and then analyzes the resulting scores by clinical significance, aspect, and error type.

The main analyses supported by the dataset are:

• Discriminative Score: the mean metric score on clinically significant pairs, defined as pairs with significance values 10–19. Lower is better, because a clinically aligned metric should penalize clinically significant errors.

• Robustness Score: the mean metric score on clinically insignificant pairs, defined as pairs with significance values 0–9. Higher is better, because a clinically aligned metric should tolerate clinically insignificant variations.

• Monotonicity: the score trajectory across the five ordered severity groups formed from the calibration subset, using significance // 4. A clinically aligned metric should generally decrease as severity increases.

A minimal example for loading and iterating through the nested JSON structure is shown below:

import json

with open("reevalmed_dataset.json", "r") as f:
    groups = json.load(f)

for group in groups:
    sig = group["significance"]      # integer from 0 to 19
    significant = sig >= 10          # binary clinical-significance split

    for pair in group["data"]:
        gt = pair["GT"]
        me = pair["ME"]
        aspect = pair["type"]              # 12 clinical aspects, or "Calibration"
        error_type = pair["error_type"]    # O / F / E / S
        explanation = pair["explanation"]

        # Example:
        # score = my_metric(reference=gt, comparison=me)
        # record score by (significant, aspect, error_type)

The monotonicity subset can be extracted from the calibration pairs:

calibration = [
    {"significance": group["significance"], **pair}
    for group in groups
    for pair in group["data"]
    if pair["type"] == "Calibration"
]

for pair in calibration:
    severity_group = pair["significance"] // 4
    # severity_group maps significance values 0–19 to groups 0–4:
    # 0–3, 4–7, 8–11, 12–15, and 16–19

The included notebook, run_metric_demo.ipynb, is a self-contained reference example. It loads reevalmed_dataset.json, scores every (GT, ME) pair with BLEU using the HuggingFace evaluate library, and prints: (i) a per-(type, error_type) table and an overall mean for the clinically significant set, corresponding to the Discriminative Score; (ii) the same table for the clinically insignificant set, corresponding to the Robustness Score; and (iii) the BLEU monotonicity trajectory across the five calibration severity groups.

BLEU is included only as a lightweight illustration of the analysis pipeline. The notebook is not intended to reproduce every metric reported in the accompanying publication. To evaluate another metric, users can replace the body of score_pair() with the desired metric call; the grouping and aggregation logic remains unchanged. Reproducing the full set of metrics in the publication requires installing and running each metric's own public implementation. No code is required simply to read or iterate through the dataset, because the JSON file is self-describing.

The dataset has several limitations. First, the rewrites are synthetic, single-aspect perturbations produced by a large language model and curated by an expert. This design isolates clinical factors for controlled evaluation, but it may not fully reproduce the distribution of errors made by real report-generation systems. Second, clinical-significance labels were validated by one experienced clinician against fixed, guidance-informed criteria rather than through independent multi-rater adjudication; therefore, the labels reflect expert judgment under a defined protocol rather than measured inter-rater consensus. Third, the resource covers chest X-ray findings only. Although the criteria were designed to be broadly applicable, they have not been validated on other imaging modalities such as CT or MRI. Fourth, the dataset is modest in scale, with 420 GT–ME pairs, because each pair requires expert validation. It is intended as a diagnostic probe for evaluation metrics, not as a large-scale training corpus [20].

The dataset is not intended for clinical diagnosis, patient care, or deployment. All annotations reflect expert judgment within a defined evaluation context and should not be interpreted as clinical guidelines. Users are responsible for adhering to the dataset license and PhysioNet policies and should clearly state that the dataset is used solely for research on evaluation methodology.

Release Notes

Version 1.0.0

Initial release of the dataset.

Ethics

This dataset is derived from MIMIC-CXR [1,2] and the Radiology Report Expert Evaluation (ReXVal) Dataset [3,4], both of which are publicly available, credentialed resources distributed on PhysioNet [5]. As a derivative work, this dataset follows the same access policy, license, and data use agreement as the parent datasets: it is distributed under the PhysioNet Credentialed Health Data License 1.5.0, and access requires the same credentialing as MIMIC-CXR, including completion of the required CITI Data or Specimens Only Research training and acceptance of the PhysioNet Credentialed Health Data Use Agreement. No new data were collected from human participants for the creation of this resource.

All source data were released under appropriate ethical approvals and data use agreements by the original providers. The present work involves secondary analysis and controlled rewriting of existing de-identified reports for research purposes only. The rewriting and annotation process modified only the clinical content of existing report text to construct ground-truth and rewritten report pairs; it did not introduce any new protected health information (PHI), and no personally identifying information was added at any stage. All derived report pairs contain only de-identified text inherited from the parent datasets, together with annotation labels and explanations.

As no identifiable human subject data were collected and no intervention or interaction with human participants was performed, additional institutional review board (IRB) approval or informed consent was not required for this study. The dataset is intended solely for research on evaluation methodology and poses minimal risk to individuals.

We believe that releasing this dataset benefits the research community by enabling more clinically aligned evaluation of medical report generation systems, while maintaining strict adherence to ethical standards for data privacy and responsible use.

Acknowledgements

We thank Zhongxin Tang, Heming Zhou, and Xin Jiang from Nanjing Medical University for their clinical consultation and discussions during the development of the clinical aspect taxonomy and significance criteria.

Conflicts of Interest

The authors declare that they have no competing interests or conflicts of interest related to this dataset.

References

1. Hartsock I, Rasool G. Vision-language models for medical report generation and visual question answering: A review. Front Artif Intell. 2024;7:1430984.
2. Bannur S, Bouzid K, Castro DC, Schwaighofer A, Thieme A, Bond-Taylor S, et al. Maira-2: Grounded radiology report generation. arXiv preprint arXiv:2406.04449. 2024.
3. Zhang Z, Lee K, Jing P, Deng W, Zhou H, Jin Z, et al. Gema-score: granular explainable multi-agent scoring framework for radiology report evaluation. In: Proceedings of the AAAI Conference on Artificial Intelligence; 2026. Vol. 40, No. 15; p. 13025-13033.
4. Zambrano Chaves JM, Huang SC, Xu Y, Xu H, Usuyama N, Zhang S, et al. A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings. Nat Commun. 2025;16(1):3108.
5. Zhao W, Wu C, Zhang X, Zhang Y, Wang Y, Xie W. Ratescore: A metric for radiology report generation. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing; 2024. p. 15004-15019.
6. Papineni K, Roukos S, Ward T, Zhu WJ. Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics; 2002. p. 311-318.
7. Lin CY. Rouge: A package for automatic evaluation of summaries. In: Text summarization branches out; 2004. p. 74-81.
8. Smit A, Jain S, Rajpurkar P, Pareek A, Ng AY, Lungren M. Combining automatic labelers and expert annotations for accurate radiology report labeling using BERT. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP); 2020. p. 1500-1519.
9. Jain S, Agrawal A, Saporta A, Truong SQ, Duong DN, Bui T, et al. Radgraph: Extracting clinical entities and relations from radiology reports. arXiv preprint arXiv:2106.14463. 2021.
10. Ostmeier S, Xu J, Chen Z, Varma M, Blankemeier L, Bluethgen C, et al. Green: Generative radiology report evaluation and error notation. In: Findings of the Association for Computational Linguistics: EMNLP 2024; 2024. p. 374-390.
11. Yu F, Endo M, Krishnan R, Pan I, Tsai A, Reis EP, et al. Radiology Report Expert Evaluation (ReXVal) Dataset (version 1.0.0). PhysioNet; 2023. Available from: https://doi.org/10.13026/2fp8-qr71
12. Yu F, Endo M, Krishnan R, Pan I, Tsai A, Reis EP, et al. Evaluating progress in automatic chest X-ray radiology report generation. Patterns. 2023;4(9):100802.
13. Li R, Li J, Jian B, Yuan K, Zhu Y. Reevalmed: Rethinking medical report evaluation by aligning metrics with real-world clinical judgment. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing; 2025. p. 11823-11837.
14. Johnson AEW, Pollard TJ, Berkowitz SJ, Greenbaum NR, Lungren MP, Deng CY, et al. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Sci Data. 2019;6(1):317.
15. Johnson A, Pollard T, Mark R, Berkowitz S, Horng S. MIMIC-CXR Database (version 2.1.0). PhysioNet; 2024. Available from: https://doi.org/10.13026/4jqj-jw95
16. Goldberger AL, Amaral LAN, Glass L, Hausdorff JM, Ivanov PC, Mark RG, et al. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation. 2000;101(23):e215–20.
17. Christensen J, Prosper AE, Wu CC, Chung J, Lee E, Elicker B, et al. ACR Lung-RADS v2022: assessment categories and management recommendations. J Am Coll Radiol. 2024;21(3):473-488.
18. Farjah F, Monsell SE, Smith-Bindman Rebecca, Gould MK, Banegas MP, Ramaprasan A, et al. Fleischner society guideline recommendations for incidentally detected pulmonary nodules and the probability of lung cancer. J Am Coll Radiol. 2022;19(11):1226-1235.
19. Leipsic J, Abbara S, Achenbach S, Cury R, Earls JP, Mancini GB, et al. SCCT guidelines for the interpretation and reporting of coronary CT angiography: a report of the Society of Cardiovascular Computed Tomography Guidelines Committee. J Cardiovasc Comput Tomogr. 2014;8(5):342-358.
20. Lu Q, Li R, Ding L, Xia Y, Zhu Y, Tao D. Beyond scalar scores: Exploring LLM-based metrics for clinical significance evaluation in radiology reports. arXiv preprint arXiv:2606.18797. 2026.