CinC Challenge 2000 Top Scores (Final)

Computers in Cardiology Challenge 2000 Top Scores (Final)

Can sleep apnea be diagnosed based on evidence from the ECG? Based on results obtained by entrants in the Computers in Cardiology Challenge 2000, yes. Each entrant has analyzed a set of 35 recordings, either for screening (in which each recording is classified as containing apnea or not), or for quantitative assessment (in which each minute of each recording is classified as containing apnea or not). Summarized below are the scores of the top finishers in each event. These are outstanding results that demonstrate conclusively that apnea can be detected reliably from the ECG using a variety of methods. (Brief descriptions of the methods used can be viewed by following the links in the tables below to abstracts submitted by each entrant for presentation at Computers in Cardiology 2000; please note that these abstracts were written prior to May 2000, and do not mention results achieved since then.) For details on the CinC Challenge 2000, follow the links at the bottom of this page.

Event 1 (Apnea Screening)

In this event, the five recordings in class B are not counted. The score is the total number of correct classifications of the 20 class A (apnea) and 10 class C (control/no apnea) recordings, so that the maximum possible score is 30. Since four entrants achieved a perfect score, the date of the top-scoring entrant's submission is the tiebreaker.

The top scores in event 1 are:

ScoreEntrantDateEntries
30/30 MR Jarvis and PP Mitra
Caltech, Pasadena, CA, USA
3 May 3
30/30 B Raymond, R Cayton, R Bates, and M Chappell
Birmingham Heartlands Hospital, Birmingham, UK
10 May 3
30/30 P de Chazal, C Henehan, E Sheridan, R Reilly, P Nolan, and M O'Malley
University College - Dublin, Ireland
17 July 1
30/30 J McNames, A Fraser, and A Rechtsteiner
Portland State University, Portland, OR, USA
12 September 3
29/30 PK Stein and PP Domitrovich
Washington University School of Medicine, St. Louis, MO, USA
12 September 2
28/30 JE Mietus, C-K Peng, PCh Ivanov, and AL Goldberger
Beth Israel Deaconess Medical Center, Boston, MA, USA (unofficial entry)
27 April 2
28/30 Z Shinar, A Baharav, and S Akselrod
Tel-Aviv University, Ramat-Aviv, Israel
29 April 1
28/30 MJ Drinnan, J Allen, P Langley, and A Murray
Freeman Hospital, Newcastle upon Tyne, UK
3 May 1
28/30 C Maier, M Bauch, and H Dickhaus
University of Heidelberg, Heilbronn, Germany
3 May 2
28/30 M Schrader, C Zywietz, V von Einem, B Widiger, G Joseph
Medical School Hannover, Hannover, Germany
7 August 8
27/30 C Marchesi, M Paoletti, S Di Gaetano
University of Firenze, Firenze, Italy
28 April 1
27/30 M Ballora, L Glass, B Pennycook, PCh Ivanov, and AL Goldberger
McGill University, Montreal, Quebec, Canada (unofficial entry)
3 May 1

Each entrant's best score is shown, along with the date when they achieved that score. Many entrants submitted multiple entries, and the 'Entries' shown indicate how many entries were submitted by each entrant up to and including the one that scored highest (later entries, and entries that did not receive scores because of formatting errors were not counted); this gives some sense of how much 'tuning' may have taken place. Entries noted as 'unofficial' came from two of the PhysioNet core research groups, and were therefore not eligible for awards, although they followed all of the rules of the competition.

Only one entry, from PK Stein, was submitted for an unofficial 'three-way score' (the total of correct classifications of all 35 records, including the 5 class B (borderline) records). This is a significantly more difficult task, since the amount of apnea in the class B records must be accurately determined in order to classify them correctly, and recognizing apnea in recordings that have only small amounts of apnea is more difficult than in recordings with frequent apneas. The single entry received a score of 33/35, an excellent result; the only errors occurred when a class A record was put in class B, and a class B record was put in class A.

Event 2 (Apnea Quantification)

In this event, each minute of each of the 35 recordings in the test set must be classified as containing apnea (A) or not (N). The maximum possible score is 17268 (the total number of minutes in the 35 recordings for which reference classifications are available).

The top scores in event 2 are:

ScoreEntrantDateEntries
15994/17268
92.62%
J McNames, A Fraser, and A Rechtsteiner
Portland State University, Portland, OR, USA
21 September 4
15939/17268
92.30%
B Raymond, R Cayton, R Bates, and M Chappell
Birmingham Heartlands Hospital, Birmingham, UK
22 September 8
15432/17268
89.36%
P de Chazal, C Henehan, E Sheridan, R Reilly, P Nolan, and M O'Malley
University College - Dublin, Ireland
22 September 15
15120/17268
87.56%
M Schrader, C Zywietz, V von Einem, B Widiger, G Joseph
Medical School Hannover, Hannover, Germany
12 September 9
15075/17268
87.30%
MR Jarvis and PP Mitra
Caltech, Pasadena, CA, USA
21 September 3
14788/17268
85.63%
Z Shinar, A Baharav, and S Akselrod
Tel-Aviv University, Ramat-Aviv, Israel
11 May 1
14772/17268
85.54%
C Maier, M Bauch, and H Dickhaus
University of Heidelberg, Heilbronn, Germany
20 September 5
14591/17268
84.49%
JE Mietus, C-K Peng, and AL Goldberger
Beth Israel Deaconess Medical Center, Boston, MA, USA (unofficial entry)
19 May 3

As in event 1, each entrant's best score is shown above, along with the date it was achieved and the number of entries submitted (excluding any entries submitted after the one that received the best score, and any that were not scored because of formatting errors). Notably, four of the top five finishers in this event also achieved perfect scores in event 1. The classification accuracy achieved by the top finishers is comparable to the roughly 90% concurrence of human experts in classification of the original polysomnograms with reference to the full set of signals, including nasal airflow, respiratory effort, and oxygen saturation.

Links: