THE EVAL_ST TOOL
--------------------------------------------------------------------------------
The EVAL_ST tool [1] (PostScript and PDF versions of the reference [1] are in the
man subdirectory - cinc04.ps and sinc04.pdf.) is an open source tool to evaluate
and compare performance and robustness of ST episode detection algorithms. The
tool supports all standard and other relevant performance measures, aggregate
gross and average statistics, and bootstrap statistical procedure to predict
real-world clinical performance. The tool (written in C) is compilable an a wide
variety of platforms and contains an additional graphic user interface module
(LessTif/Motif environment) for use on the LINUX/UNIX operating systems.
--------------------------------------------------------------------------------
INTRODUCTION
Assessing the performance and robustness of ST segment analysers and algorithms
as well as predicting their behavior in the real-world clinical environment is
a difficult task. Availability of the European Society of Cardiology ST-T
Database (ESC DB) [3], gained development of transient ST segment episode
detectors and allowed comparison of their performance. Newly developed Long-Term
ST Database (LTST DB) [2] provides a wide variety of real-world 24-hour ambulatory
records with numerous examples of transient ischemic ST segment episodes and
transient non-ischemic heart-rate related ST segment episodes. It gained further
development and evaluation of transient ST episode detectors.
Due to relative complexity of the performance measures and of the evaluation
protocol, we developed an open-source tool EVAL_ST, Version 2.0, to objective
evaluate and compare the performance and robustness of transient ST episode
detection algorithms. Since the LTST DB contains transient ischemic and transient
non-ischemic heart-rate related ST segment episodes, we initially adapted
previously developed performance measures by adding performance matrices to
differentiate between these two types of ST episodes. For detailed description
of the performance measures and of the evaluation protocol see [1] and related
references.
The tool EVAL_ST provides first- (record-by-record) and second-order (aggregate
gross and average) performance statistics for evaluation and comparison of
transient ST episode detection algorithms. The tool allows assessing the accuracy
of: 1) detecting transient ST episodes, 2) distinguishing between ischemic and
non-ischemic heart-rate related ST episodes, 3) measuring ST episode durations
and ischemic ST episode durations, and 4) measuring ST segment deviations. The
tool also provides generation of performance distributions using a bootstrap
statistical technique for predicting real-world clinical performance and
robustness.
The core module of the tool (eval_st.c) supports command-line oriented (no
graphic display) user interface style thus enabling possible batch processing.
Input to the tool are ST segment annotation streams of a reference database
(e.g., LTST DB or ESC DB) and ST segment annotation streams of the evaluated
algorithms. Evaluation results are stored to output files.
An additional graphic user interface module in LessTif/Motif graphic environment
(geval_st.c) provides graphic display of the evaluation results on LINUX/UNIX
operating system. The main window of the graphic user interface provides display
of: overlapping reference and algorithm's ST episode annotation streams of the
records, first- and second-order performance statistics, performance matrices
regarding ST episode detection and differentiation between ischemic and
non-ischemic heart-rate related ST episodes, statistics regarding ST episode and
ischemic ST episode duration detection, scatter plot of ST segment deviation
measurements, and bootstrap estimates of expected real-world performance with
performance distributions. Detailed evaluation results are stored to output files
and also graphically displayed in secondary windows of the tool.
We successfully used the tool for evaluating our two ST episode detection
algorithms [4,5] using the LTST DB and ESC DB. Besides the source code of the
EVAL_ST tool, the reference annotations of the LTST DB and ESC DB and the
annotations of the two algorithms are also provided.
Availability of the EVAL_ST tool to the world community should simplify as well
as promote easy and unique use of specific and complex performance measures and
evaluation protocol, help to easily compare different algorithms, encourage the
use of unique performance measures and evaluatio protocols in the field, and gain
the consistent use of standard performance measures.
--------------------------------------------------------------------------------
HOW TO COMPILE AND RUN THE EVAL_ST TOOL?
If you are using the Fedora, Core 2, you may wish to use the precompiled
versions of the EVAL_ST tool [1] (eval_st - command-line oriented interface,
geval_st - graphic user interface in Motif environment) which are located
in the bin directory (subdirectory to EVAL_ST directory).
After unziping and untaring the tool, entering the EVAL_ST directory, enter the
src directory. To compile the tool under Linux use the Makefile (Makefile_Linux)
script, to compile it under Solaris (UNIX) use Makefile_Motif_Solaris script. To
compile the tool, type following:
make eval_st
what will compile command-line oriented interface (eval_st.c) to eval_st.
make geval_st
will compile graphic user interface in Motif environment (geval_st.c) to
geval_st.
make
will compile both, eval_st.c and geval_st.c to eval_st and geval_st.
The source files eval_st.c and geval_st.c are equal. The only difference is
value of a constant (motif) which allows conditional compiling and is set to
0 in the eval_st.c and to 1 in the geval_st.c source file.
After that, copy the compiled tool to EVAL_ST directory:
cp eval_st ..
cp eval_st.hlp ..
cp geval_st ..
cp geval_st.hlp ..
The files eval_st.hlp and geval_st.hlp contain help texts.
After that enter the EVAL_ST directory which is working (or home directory)
for evaluation:
cd ..
To run the command-line oriented interface type:
./eval_st -[a|r] <database>_<algorithm>.evl [record] -[s|i] [-e|-b [NR_TRIALS]]
To run the graphic user interface type:
./geval_st &
Detailed description of the use of the eval_st or geval_st tool is at the bottom
of this text. Working directroy (EVAL_ST), and input and output files of the tool
are described below. Output files will appear in the EVAL_ST directory. It is
also described how to add your own evaluation project.
--------------------------------------------------------------------------------
WHAT IS IN THE EVAL_ST DIRECTORY?
The directory EVAL_ST contains evaluation project files (*.evl files) and the
subdirectories associated with this files.
The names of the evaluation project files are of the form:
<database>_<algorithm>.evl
where <database> indicates the name of the database on which an
algorithm was developed or tested and <algorithm> indicates the
algorithm.
Names of the databases are:
ltsta, ltstb and ltstc (Long-Term ST Database, LTST DB, [2]
- annotation protocols A, B and C, respectively)
esc (European Society of Cardiology ST-T Database, ESC DB, [3])
Names of the algorithms are:
alg01 (Karhunen-Loeve transform-based algorithm [4])
alg02 (Algorithm which tracks time-varying ST segment reference level
[5])
Example: To evaluate the algorithm [5] using the protocol B of the
LTST DB you select evaluation project ltstb_alg02.evl.
The subdirectories associated with the evaluation projects are of the name
<database>_<algorithm>:
<database> indicates the name of the database
<algorithm> indicates the algorithm
These subdirectories contain reference annotation files and algorithm's
annotation files for the selected pair of database and algorithm.
Currently, there are following evaluation projects files in the EVAL_ST
directory:
esc_alg01.evl
esc_alg02.evl
ltsta_alg02.evl
ltstb_alg01.evl
ltstb_alg02.evl
ltstc_alg02.evl
and following subdirectories associated with these evaluation projects:
esc_alg01
esc_alg02
ltsta_alg02
ltstb_alg01
ltstb_alg02
ltstc_alg02
In the other words, these directories contain reference annotation files and
algorithm's annotation files of the following evaluations:
1. The algorithm [4] was developed using the ESC DB [3] (evaluation project
esc_alg01.evl) and then tested using the LTST DB (protocol B) [2] (evaluation
project ltstb_alg01.evl)
2. The algorithm [5] was developed using the LTST DB (protocols A, B, and C) [2]
(evaluation projects ltsta_alg02.evl, ltstb_alg02.evl, and ltstc_alg02.evl,
respectively) and then tested using the ESC DB [3] (evaluation project
esc_alg02.evl)
The files in the <database>_<algorithm> subdirectories are following:
<record>.hea
<record>.ref
<record>.cmr
<record>.dev
<record>.alg
These files and the <database>_<algorithm>.evl file are input files to the tool.
The
<record>.hea, <record>.ref and <record>.cmr
are reference annotation files of the database (<database>),
while
<record>.alg and <record>.dev
are annotation files of the algorithm (<algorithm>). Their content is described
below.
The subdirectory esc in the EVAL_ST directory contains reference annotation
files of the ESC DB [3] only (<record>.hea, <record>.ref, and <record>.cmr).
The subdirectory ltsta in the EVAL_ST directory contains reference annotation
files of the LTST DB, protocol A, [2] only (<record>.hea, <record>.ref, and
<record>.cmr).
The subdirectory ltstb in the EVAL_ST directory contains reference annotation
files of the LTST DB, protocol B, [2] only (<record>.hea, <record>.ref, and
<record>.cmr).
The subdirectory ltstc in the EVAL_ST directory contains reference annotation
files of the LTST DB, protocol C, [2] only (<record>.hea, <record>.ref, and
<record>.cmr).
The subdirectory bin in the EVAL_ST directory contains precompiled versions of
the EVAL_ST tool (eval_st - command-line oriented interface, and, geval_st -
graphic user interface in Motif graphic environment)
The subdirectory man in the EVAL_ST directory contains article [1] in the
PostScript and PDF format, this README file, and help files (eval_st.hlp and
geval_st.hlp).
The subdirectory src in the EVAL_ST directory contains source files eval_st.c
and geval_st.c, COPYING file, makefiles, this README file, and help files.
--------------------------------------------------------------------------------
WHAT ARE INPUT FILES TO THE TOOL?
<database>_<algorithm>.evl (ASCII file)
Evaluation project file which contains names of the records of the
<database> and the path to records' location in the <database>_<algorithm>
directory when the <algorithm> is used. This file is in the working
EVAL_ST directory.
A row of this file is:
<database>_<algorithm>/<record>, i.e.,
<path_to_the_record_files>/<record>
Example:
ltstb_alg02/s20011
The files connected to each record (<record>) are following:
<record>.hea
<record>.ref
<record>.cmr
<record>.dev
<record>.alg
and are in the <database>_<algorithm> subdirectory. Their content is
described next.
<record>.hea (ASCII file, WFDB format)
The header file of the record.
<record>.ref (ASCII file)
The reference annotation stream for the record <record>. This file
contains the reference annotations for ischemic or non-ischemic
heart-rate related ST episodes as annotated by developers of the
database in each lead separately.
Each row in this file is of the form:
1. column - index of annotation [samples]
2. column - type of annotation: -1 beginnig, -2 extrema, and -3 end
of the ST episode
3. column - value 0 (at the beginning and end of an episode, if this
episode is ischemic)
3. column - value 1 (at the beginning and end of an episode, if this
episode is non-ischemic heart-rate related)
3. column - reference ST segment deviation measurement [uV] (at the
reference extrema of an episode, if this episode is ischemic
or non-ischemic heart-rate related)
4. column - lead number in which the annotation was set (0, 1, or 2)
<record>.cmr (ASCII file)
Combined reference annotation stream for the record <record>.
This file contains the reference annotations (ischemic or heart-rate
related) combined in the sense of logical OR function while ischemic
episode are more important during the combining process. It means
that having an ischemic episode in one lead and simultaneously a
heart-rate related episode in another lead, combined reference
annotation stream will contain the ischemic episode.
Each row in this file is of the form:
1. column - index of annotation [samples]
2. column - type of annotation: -1 beginning, -2 extrema, and -3 end
of the ST episode
3. column - value 0 (at the beginning and end of an episode, if this
episode is ischemic)
3. column - value 1 (at the beginning and end of an episode, if this
episode is non-ischemic heart-rate related)
3. column - reference ST segment deviation measurement [uV] (at the
reference extrema of an episode, if this episode is ischemic
or non-ischemic heart-rate related)
4. column - lead number. For the algorithm's annotation stream this
value is always 0, since this is combined annotated
stream.
<record>.dev (ASCII file)
The ST segment deviation measurements measured by the algorithm
at the reference extrema annotations. This file is similar to the
<record>.ref file, while the reference ST segment deviation
measurements are here replaced by algorithm's measurements.
Each row in this file is of the form:
1. column - index of annotation [samples]
2. column - type of annotation: -1 beginning, -2 extrema, and -3 end
of episode
3. column - value 0 (at the beginning and end of an episode, if this
episode is ischemic)
3. column - value 1 (at the beginning and end of an episode, if this
episode is non-ischemic heart-rate related)
3. column - algorithm's ST segment deviation measurement [uV] (at the
reference extrema of an episode, if this episode is ischemic
or non-ischemic heart-rate related)
4. column - lead number in which the annotation was set (0, 1, or 2)
<record>.alg (ASCII file)
The algorithm's annotation stream for the record <record>.
Each row in this file is of the form:
1. column - index of annotation [in samples]
2. column - type of annotation: -1 beginning, -2 extrema, and -3 end
of the ST episode
3. column - value 0 (at the beginning and end of an episode, if this
episode is ischemic)
3. column - value 1 (at the beginning and end of an episode, if this
episode is non-ischemic heart-rate related)
3. column - ST segment deviation measurement [uV] (at the extrema of
algorithm's ST episode - within ST deviation detection
function)
4. column - lead number. For the algorithm's annotation stream this
value is always 0 since this is combined annotated stream.
--------------------------------------------------------------------------------
WHAT ARE OUTPUT FILES OF THE TOOL?
All the output files appear in the working EVAL_ST directory.
<database>_<algorithm>_[ih|st].raw
Aggregate evaluation statistics for the algorithm <algorithm> using the
database <database>, while ischemic and non-ischemic heart-rate related
episodes are differentiated (_ih.) or are considered as episodes of
single type (_st.). This file contains numbers of annotated and detected
episodes, percentage of annotated and detected episode durations,
sensitivity (Se) and positive predictivity (+P) matrices, number of
correctly detected, falsely detected and missed episodes, and statistics
on episode detection Se/+P and episode duration detection Se/+P.
<database>_<algorithm>_[ih|st].rbr
Evaluation statistics for the algorithm <algorithm> for each record of
the database <database>, while ischemic and non-ischemic heart-rate
related episodes are differentiated (_ih.) or are considered as episodes
of single type (_st.). This file contains (for each record) a numbers
of annotated and detected episodes, percentage of annotated and detected
episode durations, Se and +P matrices, number of correctly detected,
falsely detected and missed episodes, and statistics on episode
detection Se/+P and episode duration detection Se/+P.
<database>_<algorithm>_[ih|st].ebe
Evaluation statistics of the ST segment deviation measurements at the
episode extrema, while ischemic and non-ischemic heart-rate related
episodes are differentiated (_ih.) or are considered as episodes of
single type (_st.). File contains following ST segment deviation
measurement statistics: mean [uV], standard deviation [uV], correlation
coefficient, regression line of measurements, the percentage of
measurements for which the absolute error differs by more than 100 uV,
and the value of error which 95 % of the measurements do not exceed.
Additional information is: episode number, record, lead, time of the
extrema [h:m:s], algorithm measurement [uV], reference measurement
[uV], difference between algorithm and reference measurement [uV]
and error [%].
<database>_<algorithm>_[ih|st].bts
Evaluation statistics for the bootstrap evaluation. File contains
aggregate statistics for Se and +P using the bootstrap method for the
algorithm <algorithm> and database <database>, while ischemic and
non-ischemic heart-rate related episodes are differentiated (_ih.) or
are considered as episodes of single type (_st.). The aggregate
statistics included are 5%, 16%, 84% and 95% confidence limits for
the performance statistics, minimum and maximum performance statistics,
median, mean, and difference between the mean performance statistics
and the 5% confidence limits.
<database>_<algorithm>_[ih|st].<record>
Evaluation statistics of the algorithm <algorithm> for the record
<record> of the database <database>, while ischemic and non-ischemic
heart-rate related episodes are distinguished (_ih.) or are considered
as episodes of single type (_st.). File contains numbers of annotated
and detected episodes, percentage of annotated and detected episode
durations, Se and +P matrices, numbers of correctly detected, falsely
detected and missed episodes, and statistics on episode detection Se/+P
and episode duration detection Se/+P. In addition, the file contains
also detailed matching report for each episode (reference and
algorithm's).
<database>_<algorithm>_[ih|st].smrYYY
Summary statistics, while ischemic and non-ischemic heart-rate related
episodes are differentiated (_ih.) or are considered as episodes of
single type (_st.). The YYY denotes the number of demanded summary
statistics (000-999). The file contains summary statistics of the
performed evaluations. These can include aggregate performance
statistics, bootstrap statistics and ST segment deviation measurement
statistics.
--------------------------------------------------------------------------------
HOW TO ADD YOUR OWN EVALUATION PROJECT?
In the EVAL_ST directory add your own <database>_<algorithm>.evl file and your
own <database>_<algorithm> subdirectory.
The <database>_<algorithm>.evl file should contain the paths to the record files
in the <database>_<algorithm> subdirectory.
The files in the <database>_<algorithm> subdirectory should be reference
annotation files (<record>.hea, <record>.ref and <record>.cmr) of the selected
database and annotation files of the evaluated algorithm (<record>.alg). If you
plan to evaluate also the ability to measure ST deviations, add the algorithm's
<record>.dev file.
For all these files, strictly follow the file format described above.
If your database is LTST DB, simply copy the reference annotation files from
the ltsta (protocol A), ltstb (protocol B), or ltstc (protocol C) subdirectory
(according to the selected annotation protocol) to your <database>_<algorithm>
subdirectory.
If your database is ESC DB, simply copy the reference annotation files from
the esc subdirectory to your <database>_<algorithm> subdirectory.
--------------------------------------------------------------------------------
HOW TO USE THE eval_st TOOL?
Usage of the eval_st (EVAL_ST) command-line oriented user interface to
evaluate transient ST segment episode detectors:
To write help text:
eval_st -h
To evaluate an algorithm using a single record of a database:
eval_st -r <database>_<algorithm>.evl <record> -[s|i]
<database>_<algorithm>.evl is the evaluation project file,
where <database> is database used and <algorithm> is the
algorithm evaluated
One of the options -s or -i has to be used but only one at
one time.
-r Performs evaluation of the algorithm <algorithm> using the
selected record <record> of the database <database>
-s All the episodes are considered as episodes of single type
-i Ischemic and non-ischemic heart-rate related episodes are
differentiated
Examples:
eval_st -r ltstb_alg02.evl s20011 -s
Generates performance evaluation statistics for the record s20011
while all the episodes are considered as episodes of single type
eval_st -r ltstb_alg02.evl s20011 -i
Generates performance statistics using the record s20011 for the
ischemic episodes
To evaluate an algorithm using an entire database:
eval_st -a <database>_<algorithm>.evl -[s|i] [-e|-b [NR_TRIALS]]
<database>_<algorithm>.evl is the evaluation project file,
where <database> is database used and <algorithm> is the
algorithm evaluated
One of the options -s or -i has to be used but only one at
one time. Only one of the options -e or -b can be used in one
time.
-a Performs aggregate statistics and record-by-record statistics
for the algorithm <algorithm> using the selected database
<database>
-s All the episodes are considered as episodes of single type
-i Ischemic and non-ischemic heart-rate related episodes are
differentiated
-b [NR_TRIALS] Performs the bootstrap evaluation using NR_TRIALS
bootstrap trials (default is 10000)
-e Performs evaluation of ST segment deviation measurements
Examples:
eval_st -a ltstb_alg02.evl -s
Generates aggregate statistics and record-by-record statistics
while all the episodes are considered as episodes of single type
eval_st -a ltstb_alg02.evl -i
Generates aggregate statistics and record-by-record statistics
for ischemic episodes
eval_st -a ltstb_alg02.evl -s -b
Generates aggregate statistics and record-by-record statistics,
and performs the bootstrap evaluation while all the episodes are
considered as episodes of single type
eval_st -a ltstb_alg02.evl -s -e
Generates aggregates statistics and record-by-record statistics,
and performs evaluation of ST segment deviation measurements
--------------------------------------------------------------------------------
HOW TO USE THE geval_st TOOL?
To run it, type:
./geval_st &
Usage of the geval_st (EVAL_ST) graphic user interface to evaluate
transient ST segment episode detectors:
Menu
Options Mnemonic Accelerator
Description
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
File
Open Alt+F O Ctrl+O
Opens evaluation project file <database>_<algorithm>.evl
which contain the list of records given the algorithm
<algorithm> and database for testing<database>
Examine Alt+F E Ctrl+E
Opens secondary window with information about statistics
already perfromed given the selected evaluation project
Quit Alt+F Q Ctrl+Q
Quits the interface
Evaluation_mode
Ischemic/Heart-rate ... Alt+E I Ctrl+I
Sets the evaluation mode when the ischemic and non-ischemic
heart-rate related episodes are differentiated
ST segment changes Alt+E S Ctrl+S
Sets the evaluation mode when all the episodes are considered
as episodes of single type
Single_record
Record Alt+S R Ctrl+R
Opens the dialog to select a singe record for evaluation
and then performs evaluation for the selected record
Aggregate_statistics
Evaluate Alt+A E Ctrl+A
Performs aggregate statistics for the selected evaluation
project
Extrema Alt+A X Ctrl+X
Performs evaluation of ST segment deviation measurements
for the selected evaluation project
Bootstrap
Number_of_bootstrap_trials Alt+B N Ctrl+N
Opens the dialog to set the nuber of bootstrap trials to
be used
Evaluate Alt+B E Ctrl+B
Performs the bootstrap evaluation for the selected evaluation
project
Summary
Compare Alt+M C Ctrl+C
Opens a "summary" secondary window and displays the relevant
statistics of the last three performance evaluation projects
Differences Alt+M D Ctrl+D
Comparison of algorithms
Help Help Alt+H H Ctrl+H
This message
--------------------------------------------------------------------------------
REFERENCES
[1] Jager, F., Smrdel A., and Mark, R. G. An Open Source Tool to Evaluate
Performance of Transient ST Segment Episode Detection Algorithms.
Computers in Cardiology, 2004.
[2] Jager, F., Taddei, A., Moody, G. B., Emdin, M., Antolic, G., Dorn, R.,
Smrdel, A., Marchesi, C., and Mark, R. G. Long-term ST database: a
reference for the development and evaluation of automated ischaemia
detectors and for the study of the dynamics of myocardial ischaemia.
Med. & Biol. Eng. & Comput. 2003; 41:172-182.
[3] Taddei, A., Distante, G., Emdin, M., Pisani, P., Moody, G. B.,
Zeelenberg, C., and Marchesi, C. The European ST-T database: standard for
evaluating systems for the analysis of ST-T changes in ambulatory
electrocardiography. Eur. Heart J. 1992; 13:1164-1172.
[4] Jager, F., Moody, G. B., and Mark, R. G. Detection of transient ST
segment episodes during ambulatory ECG monitoring.
Comput. Biomed. Res. 1998; 31:305-322.
[5] Smrdel, A., and Jager, F. Automated detection of transient ST-segment
episodes in 24h electrocardiograms.
Med. & Biol. Eng. & Comput. 2004; 42:303-311.
--------------------------------------------------------------------------------