VitalDB - AnesLLM: a dataset and benchmark based on real intraoperative anaesthetic decisions 1.0

File: <base>/docs/evaluation_guide.md (3,956 bytes)
# AnesLLM — Evaluation Guide

This guide explains how to load the dataset and evaluate a model against it.

## 1. Loading the data

The data lives in `dataset/data/{dev,train,test}/case<caseid>.jsonl`: one JSON object per line, one
line per decision window, all windows of a case in the same file.

```python
import json
from pathlib import Path

def load(split):
    for f in sorted((Path("dataset/data") / split).glob("case*.jsonl")):
        for line in open(f, encoding="utf-8"):
            yield json.loads(line)

for rec in load("test"):
    x = rec["input"]      # dict with 280 pre-decision features
    out = rec["output"]   # dict with result_1 / result_aux / result_real
```

## 2. Record contract

```json
{
  "window_id": "case14_w127",
  "case_id": "14",
  "input": { "...280 fields, everything observable BEFORE the decision..." },
  "output": {
    "result_1":    { "action": "reduce_hypnotic", "ratio": 0.48 },
    "result_aux":  { "action": "reduce_opioid",   "ratio": 0.45 },
    "result_real": "reduce_hypnotic"
  }
}
```

- **`input`** is the model input. Nulls are encoded as JSON `null`. The nested `*_json` fields
  are JSON objects (e.g. `vitals_json`, `staleness_json`, `actions_history_json`).
- **`output.result_real`** is the ground-truth action (no ratio) — the action the clinician
  actually performed.
- **`output.result_1`** and **`output.result_aux`** are the primary and alternative consensus
  actions: top-2 of a LightGBM model trained to predict the clinician's action (case-grouped
  out-of-fold predictions); `ratio` is the model probability.

## 3. Action vocabulary

Ground truth (`result_real`) is always one of:

- `increase_hypnotic`, `reduce_hypnotic`, `increase_opioid`, `reduce_opioid`, `no_action`

The consensus alternatives (`result_1`, `result_aux`) may additionally be `vasopressor` (a valid
*option*, but never a ground-truth label). Ventilation actions never appear.

## 4. Metrics

Two standard metrics capture how well a model reproduces the clinician:

- **top-1 accuracy** — the model's single best action equals `result_real`:
  `acc = mean(argmax(model(x)) == result_real)`.
- **top-2 recall** — `result_real` is among the model's two best actions.

Reference numbers — agreement between the generated alternatives and the real action, on the
full set:

| metric | value |
|---|---|
| top-1 accuracy | 55.8% |
| top-2 recall | 80.4% |

Because the true action is not always unique (several actions can be acceptable), the dataset
also provides the alternative action and supports external-validity evaluation (see
`docs/dataset_design.md`, §5–6).

## 5. Difficulty

Each case has a difficulty label in `dataset/reports/split_manifest.csv`
(`case_id, difficulty, split, n_windows, null_score`):

- `easy` — high consensus confidence (≥ p75 of the per-case confidence),
- `hard` — low consensus confidence (< p25),
- `medium` — otherwise.

Use these to report performance by difficulty (a model should do better on `easy` than on
`hard`).

## 6. Splits

- **`train`** — train your model here.
- **`test`** — balanced (stratified by difficulty) and curated to have **no physiologically
  impossible values**; the reference for final evaluation.
- **`dev`** — 25 edge cases with heavy missing data and deliberately retained bad values; use it
  to harden your data-loading and null handling, not for performance reporting.

## 7. Practical notes

- **Nulls**: coverage varies a lot per variable (see `dataset/reports/coverage_by_variable.csv`).
  The `dev` split is the stress test for null handling.
- **Vitals**: current values are also available as top-level `*_current` fields; the raw
  time series are in `input.vitals_json` (signals: MAP, HR, BIS, SBP, DBP, SpO2, EtCO2).
- **No leakage**: `input` contains no post-decision information (audited — see
  `docs/audit_report.md`).
- **Identifiers**: drop `pt_caseid` and `pt_subjectid` before training or release.