CinC Challenge 2001 Top Scores (Final)

Computers in Cardiology Challenge 2001 Top Scores (Final)

The top scorers in the 2001 challenge were announced during the 25 September plenary session of Computers in Cardiology in Rotterdam. The top score and the award in event 1 was obtained by Günther Schreier and colleagues of the Austrian Research Centers Seibersdorf (Graz, Austria). In event 2, the top score was obtained by Wei Zong and colleagues at the Harvard-MIT Division of Health Sciences and Technology (Cambridge, Massachusetts, USA); because this entry came from one of the PhysioNet core research groups, however, it was unofficial, and the award in event 2 was given to the team with the highest official score, who were once again Günther Schreier and colleagues. We congratulate and thank all of the participants in this challenge.


Wei Zong and Günther Schreier receive congratulations from challenge organizer George Moody at CinC 2001.
(Photo courtesy of Bill Sanders.)

The immediate goal of the Computers in Cardiology Challenge 2001 was to develop a fully automated method to predict the onset of paroxysmal atrial fibrillation/flutter (PAF), based on the ECG prior to the event. Entrants developed and tested methods using a database created for this challenge.

The test set of the PAF Prediction Challenge Database consists of 50 pairs of half-hour ECG recordings. Each pair of recordings is obtained from a single 24-hour ECG. Subjects in group A experienced PAF; for these subjects, one recording ends just before the onset of PAF, and the other recording is distant in time from any PAF (there is no PAF within 45 minutes before or after the excerpt). Subjects in group N do not have PAF; in these, the times of the recordings have been chosen at random. Entrants were told that between 20 and 30 of the 50 subjects belong to each group; the exact size of each group (28 in group A, 22 in group N) was disclosed only after the conclusion of the challenge in September.

Brief descriptions of the methods used can be viewed by following the links in the tables below to abstracts submitted by many of the entrants for presentation at Computers in Cardiology 2001; please note that these abstracts were written no later than May 2001, and do not mention results achieved since then. For details on the CinC Challenge 2001, follow the links at the bottom of this page.

Event 1 (PAF Screening)

Event 1 was intended to determine if subjects in group A can be distinguished from those in group N. (In other words, can individuals at risk of PAF be identified within a larger population, based on their ECGs?) The number of correctly classified subjects (0 to 50) is the event 1 score. The best methods were able to achieve roughly 80% classification accuracy on the test set.

The top scores in event 1 are:

ScoreEntrantDateEntries
41/50
82%
G Schreier, P Kastner, and W Marko
Austrian Research Centers Seibersdorf, Graz, Austria
17 September 8
40/50
80%
W Zong and RG Mark
Harvard-MIT Division of Health Sciences and Technology, Cambridge, MA, USA (unofficial entry)
12 September 7
37/50
74%
R Sweeney and colleagues
Guidant Corp., St. Paul, MN, USA
8 May 3
36/50
72%
C Maier, M Bauch, and H Dickhaus
University of Applied Sciences, Heilbronn, Germany
19 September 2
35/50
70%
C Marchesi and M Paoletti
Università di Firenze, Firenze, Italy
27 April 1
34/50
68%
KS Lynn and HD Chiang
Cornell University, Ithaca, NY, USA
28 April 6
33/50
66%
CC Yang
National Yang-Ming University, Taipei, Taiwan
21 April 4
33/50
66%
JA Kors
Erasmus University, Rotterdam, The Netherlands
10 July 2
32/50
64%
P de Chazal and C Heneghan
University of New South Wales, Sydney, Australia
13 September 1
32/50
64%
R Loesch 14 September 6

Each entrant's best score is shown, along with the date when they achieved that score. Many entrants submitted multiple entries, and the 'Entries' shown indicate how many entries were submitted by each entrant up to and including the one that scored highest (later entries, and entries that did not receive scores because of formatting errors were not counted); this gives some sense of how much 'tuning' may have taken place. The entry noted as 'unofficial' came from one of the PhysioNet core research groups, and was therefore not eligible for awards, although the entrant followed all of the rules of the competition.

Event 2 (PAF Prediction)

Event 2 was intended to determine if subjects in group A have distinctive and detectable changes in their ECGs immediately before PAF. (In other words, is the imminent onset of PAF predictable in an individual known to be at risk of PAF?) A successful method for doing so should be able to determine which record of each pair of group A records immediately precedes PAF. If the identities of the group A records were known, it would be sufficient to classify these records only; since the goal of event 1 was to identify group A, however, we did not provide this information! Entrants in event 2 of the challenge therefore were required to classify exactly one of each pair of records in the test set as `A' (defined as ``immediately preceding PAF, if the patient belongs to group A''), and the other as `N' (defined as ``not immediately preceding PAF''). One point was awarded for each correctly classified record pair, so that the raw event 2 scores range from 22 to 50 (the lower bound is 22, the number of subjects in group N, because the group N subjects are always considered correctly classified). In the table below, the scores have been adjusted by subtraction of 22 from the raw scores, so that the adjusted scores can range between 0 and 28.

The top scores in event 2 are:

ScoreEntrantDateEntries
22/28
79%
W Zong and RG Mark
Harvard-MIT Division of Health Sciences and Technology, Cambridge, MA, USA (unofficial entry)
1 May 1
20/28
71%
G Schreier, P Kastner, and W Marko
Austrian Research Centers Seibersdorf, Graz, Austria
19 August 2
19/28
68%
P de Chazal and C Heneghan
University of New South Wales, Sydney, Australia
28 April 1
19/28
68%
C Maier, M Bauch, and H Dickhaus
University of Applied Sciences, Heilbronn, Germany
11 September 3
18/28
64%
KS Lynn and HD Chiang
Cornell University, Ithaca, NY, USA
29 April 2
17/28
61%
P Langley, D di Bernardo, J Allen, E Bowers, F Smith, S Vecchietti, and A Murray
Freeman Hospital, Newcastle upon Tyne, UK
30 April 1
17/28
61%
D Gamberger and T Smuc
Rudjer Boskovic Institute, Zagreb, Croatia
23 August 2
16/28
57%
CC Yang
National Yang-Ming University, Taipei, Taiwan
23 April 1
16/28
57%
R Sweeney and colleagues
Guidant Corp., St. Paul, MN, USA
8 May 1
15/28
54%
L Almarro
UPV, Valencia, Spain
30 April 1

As in event 1, each entrant's best score is shown above, along with the date it was achieved and the number of entries submitted (excluding any entries submitted after the one that received the best score, and any that were not scored because of formatting errors).

Links: