# AnesLLM Dataset — Audit Report This report describes the verification checks performed on the final `dataset/` (dev/train/test) and their results. It is purely descriptive; the corresponding machine-readable outputs live in `dataset/reports/`. ## 1. Split integrity Checks that the three splits are disjoint and well-formed. - **Case overlap** between dev/train/test: **0** — every case appears in exactly one split. - **Duplicate `window_id`**: **0**. - **Temporal order**: 0 cases with decreasing timestamps. 526 cases have two consecutive windows with the *same* `time_since_induction_sec` (duplicate timestamps, not disorder). Output: `dataset/reports/split_integrity.csv`. ## 2. Output consistency Checks the validity of the `output` block of every window. - `ratio` values within `[0, 1]` and `result_1.ratio + result_aux.ratio ≤ 1`: **0 violations**. - `result_1.action ≠ result_aux.action`: **always true**. - Vocabulary: `result_real` always in the 5 ground-truth classes; `result_1`/`result_aux` always in the 6-option vocabulary (5 classes + `vasopressor`): **0 violations**. - Concordance (agreement between model and clinician): | Group | top1 = result_real | result_real ∈ top2 | |---|---|---| | overall | 55.8% | 80.4% | | easy | 61.5% | 86.6% | | medium | 56.5% | 80.9% | | hard | 50.1% | 74.7% | Output: `dataset/reports/output_consistency.csv`. ## 3. Leakage Confirms no post-decision information is present in `input`. All 21 excluded columns (`gt_action`, `event_*`, `acceptable_set`, `top2*`, `entropy`, `fragility`, `dt_next`, horizons `*_h1..h5`, presence flags, etc.) are absent from `input`: **no leakage**. Output: `dataset/reports/leakage.csv`. ## 4. Class balance Ground-truth class distribution (overall): | action | share | |---|---| | `no_action` | 27.9% | | `reduce_opioid` | 27.8% | | `increase_opioid` | 24.3% | | `reduce_hypnotic` | 12.1% | | `increase_hypnotic` | 8.0% | Output: `dataset/reports/class_balance.csv`. ## 5. Numeric sanity Range checks against physiologically plausible bounds. Most variables are clean. Findings: - **Impossible hemodynamic values** (negative or absurd blood pressure, e.g. `MAP < 0` or `MAP > 250`) affect **1,730 windows** across dev and train, originating from VitalDB track artifacts. They are **kept but reported** (see §9). - `lab_cr`: 27 values above 20 mg/dL (max 21.45). `mac`: 1 value above 3 (max 3.1). - `PPV`/`SVV` showed heavy artifacts (>100%) and were **removed entirely** from the dataset. Output: `dataset/reports/numeric_sanity.csv`. ## 6. Train/test drift Compares feature distributions between `train` and `test` (Kolmogorov–Smirnov test + Population Stability Index) to verify the test set is representative. - `result_real` class distribution: total-variation distance **0.011** → no drift. - No feature shows meaningful drift except `lab_k` (PSI 0.14) and `lab_plt` (PSI 0.11), mildly shifted. The dataset is **portable** between train and test. Output: `dataset/reports/train_test_drift.csv`. ## 7. Missingness Null fraction per window, aggregated by split and difficulty. | Group | mean null score | |---|---| | dev | 0.313 | | train | 0.119 | | test | 0.124 | | easy | 0.129 | | medium | 0.117 | | hard | 0.120 | Output: `dataset/reports/missingness.csv`. ## 8. Coverage per variable Percentage of non-null values for each of the 280 input fields (plus the 7 vital signals), broken down by split and by difficulty. Output: `dataset/reports/coverage_by_variable.csv`. ## 9. Bad-value case inventory Cases containing at least one physiologically impossible value, with the offending fields: - `dev`: 1 case (`case3807`, 5/423 windows = 1.18%). - `train`: 864 cases (1,725/80,933 windows = 2.13%). - `test`: **0** — any case with impossible values was moved out of test into train to guarantee maximum evaluation quality. Output: `audit/bad_cases.csv` (case, split, number of bad windows, reasons). ## 10. Privacy (k-anonymity) Evaluated at window level (quasi-identifiers: sex, age, weight, height, BMI, ASA, with and without the current vitals HR/MAP/SpO2/BIS/EtCO2; quantitative values rounded to integers). - With **demographics only**, every window belongs to a group of **k >= 3** (min k = 3, median k = 23), but this is misleading: windows of the same patient repeat the same profile. - Adding **vital signs** makes **99.6% of windows unique (k = 1)**: the concrete vitals re-identify the window. - Direct identifiers `pt_caseid` and `pt_subjectid` are present and must be removed before any public release. Output: `dataset/reports/k_anonymity.csv`. ## 11. Data dictionary Every variable is documented (identifier, type, English description) in `dataset/reports/data_dictionary.csv` (292 entries: 280 input fields + 7 vitals + 5 output fields).