Searching for Premature Ventricular Contraction and Supraventricular Premature Beat from Long-term ECGs: The 3rd China Physiological Signal Challenge 2020
If you use the Challenge data for paper publication, please cite this paper for Challenge data description:
Z.P. Cai, C.Y. Liu*, H.X. Gao, X.Y. Wang, L.N. Zhao, Q. Shen, E. Y. K. Ng, and Jianqing Li*. An Open-Access Long-Term Wearable ECG Database for Premature Ventricular Contractions and Supraventricular Premature Beat Detection. Journal of Medical Imaging and Health Informatics, 2020, 10(11): 2663-2667.
Invitation Letter (in English) Invitation Letter (in Chinese)
Welcome to join CPSC 2020 Wechat Group for free discussion and information sharing. Please add the conference wechat and remark with "Please invite me into the Challenge Wechat Group"
News
The final challenge scores of CPSC 2020 are shown as below, with the final code available as open-source data.Code No. | Institution/Affiliation | Team Members | Score PVC | Score SPB |
CPSC1077 | Shinall Technology | Min Chen, Kui Dong | 41479 | 92947 |
CPSC1091 | University of Shanghai for Science and Technology | Wenjie Cai, Jingying Yang, Jianjian Cao, Xuan Wang | 55706 | 120942 |
CPSC1093 |
1. Soochow University; 2. Suzhou Institute of Biomedical Engineering and Technology Chinese Academy of Sciences |
Lirong Wang1, Lishen Qiu2, Wenqiang Cai1, Wenliang Zhu2, Jie Yu1, Wanyue Li1, Duoduo Wang1, Huimin Zhang1 | 95900 | 111523 |
CPSC1082 | Beijing University of Technology | Minggang Shao, Zhuhuang Zhou, Shuicai Wu | 97913 | 95348 |
CPSC1089 | Chengdu Spaceon Electronics CO., LTD. | Shan Yang, Chunli Wang, Heng Xiang, Qingda Kong | 142228 | 117942 |
CPSC1104 | Tsinghua University | Hao Wen | 143484 | 99824 |
CPSC1085 | Taiwan AI Academy; Academia Sinica; National Taiwan University | Tsai-Min Chen | 144966 | 153040 |
CPSC1098 | Northeastern University | Yan Li, Yuxiang Li, Haixu Yang, Jihong Liu | 151735 | 215664 |
CPSC1092 | Harbin Institute of Technology | Yang Liu, Runnan He | 166215 | 160474 |
CPSC1081 | Institute of Semiconductors, Chinese Academy of Sciences; University of Chinese Academy of Sciences | Yibo Yin; Sitao Zhang | 168578 | 195467 |
CPSC1088 | East China Jiaotong University | Feng Mei, Qian Hu, Lingfeng Liu | 362348 | 120410 |
[2020-10-16] New Sample Python entry has been updated to avoid repeated counting false posibility(FP).
[2020-8-31] Sample Python entry has been updated for issues in FP statistics.
[2020-8-18] In response to the training set label problem raised by some contestants, we confirmed and updated all the labels, please download the new Training Set for your code preparation.
[2020-8-12] It is necessary to to provide detailed descriptions and version numbers of dependent third-party libraries in the form of requirements.txt. It is recommended to apply for tensorflow>=1.13.1, or tensorflow>=2.0, keras>=2.2.4,pytorch.
[2020-8-12] Sample Python entry has been updated for cross-system issues.
The Score List has been indicated as below, more will be updated soon…
Code No. | Institution/Affiliation | Team Members | ScorePVC | ScoreSPB |
CPSC841 | Shinall Technology | Min Chen, Kui Dong | 12246 | 36640 |
CPSC775 | Beijing University of Technology | Minggang Shao, Zhuhuang Zhou, Shuicai Wu | 15062 | 77440 |
CPSC1054 | University of Shanghai for Science and Technology | Wenjie Cai, Jingying Yang, Jianjian Cao, Xuan Wang | 21285 | 50941 |
CPSC982 | Harbin Institute of Technology | Yang Liu, Runnan He | 23654 | 101026 |
CPSC769 | Chengdu Spaceon Electronics CO., LTD. | Shan Yang, Chunli Wang, Heng Xiang, Qingda Kong | 33597 | 73773 |
CPSC1051 | Northeastern University | Yan Li, Yuxiang Li, Haixu Yang, Jihong Liu | 40152 | 124095 |
CPSC1033 |
1. Soochow University; 2. Suzhou Institute of Biomedical Engineering and Technology Chinese Academy of Sciences |
Lirong Wang1, Lishen Qiu2, Wenqiang Cai1, Wenliang Zhu2, Jie Yu1, Wanyue Li1, Duoduo Wang1, Huimin Zhang1 | 45785 | 74590 |
CPSC897 | Shenzhen Soonview Technology Co., Ltd. | Longbo Zhang, Chi Wang | 62955 | 79895 |
CPSC917 | Taiwan AI Academy; Academia Sinica; National Taiwan University | Tsai-Min Chen | 63192 | 80772 |
CPSC950 | Shanghai National Group Health Technology Co., Ltd | Jie Yang | 63187 | 80772 |
CPSC664 | Institute of Scientific Instruments of the CAS Brno | Adam Ivora | 76290 | 87382 |
CPSC758 |
Xi'an Jiaotong University; The University of Manchester; University of Oxford |
Jingyao Wu, Zhibin Zhao, Ruqiang Yan; David C. Wong; Nicola K Dinsdale |
89310 | 77900 |
CPSC829 | Tsinghua University | Hao Wen | 94962 | 53300 |
CPSC1027 | Institute of Semiconductors, Chinese Academy of Sciences; University of Chinese Academy of Sciences | Yibo Yin, Sitao zhang | 88257 | 98543 |
CPSC933 | East China Jiaotong University | Feng Mei, Qian Hu, Lingfeng Liu | 141159 | 165949 |
CPSC921 | Northeastern University | Haixu Yang, Yan Li, Jihong Liu | 169794 | 253054 |
CPSC731 | Nalong Technology | Xi Li | 180725 | 78890 |
CPSC722 | Soochow University | Yifan Li | 227492 | 100140 |
CPSC748 | Qatar University; Izmir University of Economics | Muhammad Uzair Zahid, Ozer Can Devecioglu, Turker Ince, Mustafa Serkan Kiranyaz | 227650 | 98740 |
CPSC802 | Zhejiang University | Shao | 227650 | 100185 |
CPSC794 | Universidad Nacional Autónoma de México | Jonathan Roberto Torres Castillo, Karen Patricia Gaitan de lo Rios, Miguel Angel Padilla Castañeda | 227650 | 100185 |
CPSC727 | Lenovo Group Ltd. | Ping Xu | 227650 | 100185 |
CPSC953 | Sichuan University | Lei Xiong, Leiliang Qin | 227650 | 100185 |
CPSC766 | University of Electronic Science and Technology of China | Jin Qi, Sunfeng Luo, Xiaofang He | 227650 | 100185 |
CPSC789 | Ludong University | Shengqiang Liu, Laiguo Li, Qingran Dong, Shuo Zhu, Lijun Guo, Wenxing Shi, Qian Li, Xiaochen Wei, Jing Li | 227650 | 100634 |
CPSC797 | Ludong University | Aowei Liu, Zixuan Liu | 227651 | 100183 |
Frequently Asked Questions (FAQs)
Q1: How can we register team information to join the challenge?R1: We suggest that the team leader, or one of the team representatives registers an account on behalf of the team in the system. Please DO NOT register multiple accounts from one team.
Q2: How should I submit the challenge entries with the provided sample entry ‘CPSC2020_challenge’?
R2: You can add the signal processing codes in the ‘CPSC2020_challenge.m', and submit the updated version of 'CPSC2020_challenge.m'. At this moment, please focus on the methods you need to develop, and submit the ‘CPSC2020_challenge.m’ with developed codes. Test data and training data are exactly in the same format.
Q3: How should I name the main function?
R3: Please name it CPSC2020_challenge.
Q4: If we use the python environment, can the test system support commonly used packages, such as keras, wfdb, numpy?
R4: Yes, It can.
Q5: How can I contact the organizing committee?
R5: Please join the Wechat Group or email to chinachallenge_cpsc@icbeb.org.
Introduction
Abnormality of cardiac conduction system can induce arrhythmia. Abnormal heart rhythm can lead to other cardiac diseases and complications, and can be life-threatening [1]. There are various types of arrhythmias and each type is associated with a pattern, and as such, it is possible to be identified. Arrhythmias can be classified into two major categories. The first category consists of arrhythmias formed by a single irregular heartbeat in electrocardiogram (ECG), herein called morphological arrhythmia, while another category consists of arrhythmias formed by a set of irregular heartbeats in ECG, herein called rhythmic arrhythmias [2]. Dynamic electrocardiogram (DCG), like ECG Holter, provides an important way to monitor the incidences of arrhythmias in daily life, facilitating the doctors to check a total number and distribution of arrhythmias in a long time and thus to provide the required therapy to prevent further problems.The 3rd China Physiological Signal Challenge 2020 (CPSC 2020) aims to encourage the development of algorithms for searching for premature ventricular contraction (PVC) and supraventricular premature beat (SPB) from 24-hour dynamic single-lead ECG recordings usually with low signal quality and/or abnormal rhythm waveforms. Similar the previous works and efforts of the CPSC 2018 [3] and CPSC 2019 [4], accurate locating of abnormal heartbeats is another critical issue put forward here for further discussion.
ECG signal provides an important role in non-invasively monitoring and clinical diagnosis for cardiovascular disease (CVD). Arrhythmia detection is one of the ultimate goals of routine ECG monitoring, and PVC and SPB are the two most common arrhythmias. Increase in these beats may be a precursor to stroke or sudden cardiac death [5]. Although their detection methods have been severely tracked throughout the last several decades, accurate and robust detections are still challenging in noisy or low-signal quality environment, especially for daily monitored ECG waveforms. It is true that many of the developed PVC and SPB detection algorithms can achieve high accuracy (over 96% in sensitivity and positive predictivity) when tested over the standard ECG databases such as the MIT-BIH Arrhythmia Database or AHA Database [6]. However, these algorithms may fail when used in the noisy environment. Especially, even the basic QRS detection can be invalid in the low signal quality ECG analysis [7]. A recent study confirmed that none of the common QRS detection algorithms can obtain 80% detection accuracy when tested in a dynamic noisy ECG database. In this year’s challenge, we provide a new ECG database containing long-term noisy ECG recordings from clinical arrhythmia patients, to encourage the participants to develop more efficient and robust algorithms for PVC and SPB detection.
Challenge Data
Training data consists of 10 single-lead ECG recordings collected from arrhythmia patients, each of the recording last for about 24 hours (shown in Table 1). Table 1 also indicates the patient if he/she is an atrial fibrillation (AF) patient. Test set contains similar ECG recordings, which is unavailable to public and will remain private for the purpose of scoring for the duration of Challenge and for some period afterwards. All data were collected by a unified wearable ECG device with a sampling frequency of 400 Hz, and provided in MATLAB format (each including three *.mat file: one is ECG data and another two are the corresponding PVC and SPB annotation files, respectively).Recordings | AF patient ? | Length (h) | # N beats | # V beats | # S beats | # Total beats |
A01 | No | 25.89 | 109,062 | 0 | 24 | 109,086 |
A02 | Yes | 22.83 | 98,936 | 4,554 | 0 | 103,490 |
A03 | Yes | 24.70 | 137,249 | 382 | 0 | 137,631 |
A04 | No | 24.51 | 77,812 | 19,024 | 3,466 | 100,302 |
A05 | No | 23.57 | 94,614 | 1 | 25 | 94,640 |
A06 | No | 24.59 | 77,621 | 0 | 6 | 77,627 |
A07 | No | 23.11 | 73,325 | 15,150 | 3,481 | 91,956 |
A08 | Yes | 25.46 | 115,518 | 2,793 | 0 | 118,311 |
A09 | No | 25.84 | 88,229 | 2 | 1,462 | 89,693 |
A10 | No | 23.64 | 72,821 | 169 | 9,071 | 82,061 |
Although PVC and SPB detection algorithms are widely studied for many years, accurate detection is still challenging in this Challenge due to the amplitude variation, morphological variation, as well as noises in ECG data. Figure 1 shows examples of ECG waveforms in training data.
Figure 1. Examples of demonstrated ECG waveforms. Black circles denote the reference locations of SPB and red ones denote the reference locations of PVC. The challenges for accurate PVC and SPB detection are from: A) amplitude variability and multi-source PVC; B) noise and other abnormal interference (such as AF) and C) electrode sliding interference.
Challenge Events and Scoring
CPSC 2020 is comprised of two events related to scoring: PVC detection and SPB detection. PVC and SPB annotations in the training and test sets are labeled and initially confirmed by cardiologists and trained volunteers. Score is calculated according to the following rules.Event 1: PVC detection
In this event, the goal is to generate a set of PVC annotations for each recording that can matches the reference PVC annotations. For each reference PVC annotation, a matched PVC annotation should lie in 150 ms duration centered by the reference PVC annotation [8]. Noted that the reference PVC annotations appear in the first and last 0.2 seconds are ignored. Detected PVC should be within 150 ms from the reference ones. The scoring rules are:
• a false positive ( FP) detection deduct 1 point.
• a false negative ( FN) detection deduct 5 points, since from a clinical perspective, missed diagnosis is more serious than misdiagnosis, thus we penalize FN detection.
The final score for Event 1 ( PVC err) is the sum of all deducted points.
Event 2: SPB detection
In this event, the goal is to generate a set of SPB annotations for each recording that can matches the reference SPB annotations. For each reference SPB annotation, a matched SPB annotation should lie in 150 ms duration centered by the reference SPB annotation [8]. Noted that the reference SPB annotations appear in the first and last 0.2 seconds are also ignored. Detected SPB should be within 150 ms from the reference ones. The scoring rules are:
• a false positive ( FP) detection deduct 1 point.
• a false negative ( FN) detection deduct 5 points, since from a clinical perspective, missed diagnosis is more serious than misdiagnosis, thus we penalize FN detection.
The final score for Event 2 ( SPB err) is the sum of all deducted points.
Quick Start
1. Download the training set[2020-8-18] and the sample entry ( Sample MATLAB entry/ Sample Python entry[2020-8-12] ).2. Create a free account and join the CPSC Challenge 2020 project.
3. Develop your entry by editing the existing files:
• You are allowed modify the sample entry source code file CPSC2020_challenge.m with your changes and improvements. But, never alter any word in CPSC2020_score.m to avoid a wrong result.
• Modify the AUTHORS.txt file to include the names of all the team members.
• Run your modified source code file on all the records in the training set by executing the script CPSC2020_score.m. This will also build a new version of entry.zip.
• Optional: Include a file named DRYRUN in the top directory of your entry (where the AUTHORS.txt file is located) if you do not wish your entry to be scored and counted against your limit. This is useful in cases where you wish to make sure that the changes made do not result in any error.
4. Please note: we cannot guarantee the 100% accuracy for the reference PVC/SPB annotations. If you find that the reference is questioned (error), please give us a feedback .txt file using the following format to us:
File name: "data label name - event type (S / V) - sampling point position - expected result" (example: A07-S-76214-N.txt). Please keep the ‘.txt’ content empty.
We will further verify this with the cardiologists, and update the reference annotations in the website periodically.
Participating in the challenge
To participate in the challenge, you need to submit a software written by MATLAB or Python that is able to run on the test set and output the final scoring result without user interaction in our test environment. One sample entry CPSC2020_challenge.m is available to help you get started. And a scoring function CPSC2020_score.m is provided for competitors testing algorithms and build a new version of entry.zip. The team must submit the first challenge open-source entry before July 1, 2020. The deadline for submitting the official challenge entries is October 25, 2020.Awards and Rules
The winner will be selected on the basis of the obtained final PVC err and SPB err on the hidden test data. The first three for each Event challenging will receive certificates and generous bonuses:• First prize: Certificate plus bonus of RMB 15,000
• Second prize: Certificate plus bonus of RMB 10,000
• Third prize: Certificate plus bonus of RMB 5,000
We welcome all the individual or research group around the world to attend the challenge. To be eligible for the open-source award, please do the following (see table 2):
(1) Register the team before June 1, 2020.
(2) Submit at least one open-source entry that can be scored before July 1, 2020.
(3) Attend ICBEB 2020 (November 15-18, 2020) and present your work there.
All deadlines occur at 11:59pm GMT+8 (UTC) on the dates mentioned below. If you do not know the difference between GMT and your local time, find out what it is before the deadline!
Start | Entry limit | End | ||||
Registration | 1 February | At least once | 30 June | |||
First Stage | 1 July | 10 | 31 September | |||
Second Stage | 1 October | 3 | 25 October | |||
Final Stage | 26 October | Final entry | 1 November | |||
In the interest of fairness to all participants, any entries received later than deadline of that stage will not be accepted or scored. Code submitted more than the limit number of times will not be scored.
Please do not submit analysis of this year’s Challenge data to other Conferences or Journals until after ICBEB 2020 has taken place, so the competitors are able to discuss the results in a single forum.
Important Dates
15 February, 2020 -- Challenge open
June 1, 2020 -- Deadline for team registration to join the challenge
July 1, 2020 -- Deadline for submitting the first challenge open-source entry
October 25, 2020 -- Deadline for submitting the challenge entries
November 1, 2020 -- Confirm for submitting the best entry for final scoring
November 15-18, 2020 – Announcement of the winners of CPSC 2020 in ICBEB 2020
Any questions or problems about the Challenge, please feel free to contact chinachallenge_cpsc@icbeb.org
Challenge Chair:
Prof. Chengyu Liu, Southeast University, China
Challenge Committee:
Dr. Zhipeng Cai, Southeast University, China
Dr. Hongxiang Gao, Southeast University, China
Dr. Xingyao Wang, Southeast University, China
Dr. Qin Shen, Nanjing Medical University, China
Dr. Xiangwei Ding, Nanjing Medical University, China
International Advisory Chair:
Prof. Gari D. Clifford, Emory University & Georgia Institute of Technology, USA
International Advisory Co-chairs:
Prof. Aiguo Song, Southeast University, China
Prof. Jianqing Li, Nanjing Medical University, China
Prof. Zuhong Lu, Southeast University, China
Prof. Yi Peng, Peking Union Medical College, China
Mr. Yingjia Yao, Lenovo Group, China
International Advisory Committee:
Prof. Eddie Ng Yin Kwee, Nanyang Technological University, Singapore
Prof. Kang-Ping Lin, Chung Yuan Christian University, Chinese Taipei
Prof. Feng Wang, Southeast University, China
Prof. Zhengtao Cao, Air Force Medical Center, China
Prof. Xianzheng Sha, China Medical University, China
Prof. Xingming Guo, Chongqing University, China
Prof. Shoushui Wei, Shandong University, China
Prof. Zhi Tao, Suzhou University, China
Dr. Alistair Johnson, MIT, USA
Hosted by:
School of Instrument Science and Engineering, Southeast University, China
The State Key Laboratory of Bioelectronics, Southeast University, China
School of Biomedical Engineering and Information, Nanjing Medical University, China
Supported by:
ICBEB Organizing Committee
Health Engineering Committee of Chinese Society of Biomedical Engineering
Biomedical Sensor Committee of Chinese society of Biomedical Engineering
Youth Working Committee of Chinese Society of Biomedical Engineering
Jiangsu Instrument and Control Society
Awards sponsored by:
Lenovo Group
Reference
[1] S. L. Oh, E. Y. Ng, R. San Tan, and U. R. Acharya, "Automated diagnosis of arrhythmia using combination of CNN and LSTM techniques with variable length heart beats," Computers in biology and medicine, vol. 102, pp. 278-287, 2018.
[2] E. J. D. S. Luz, W. R. Schwartz, G. Cámara-Chávez, and D. Menotti, "ECG-based heartbeat classification for arrhythmia detection: A survey," Computer methods and programs in biomedicine, vol. 127, pp. 144-164, 2016.
[3] F. Liu, C. Liu, L. Zhao, X. Zhang, X. Wu, X. Xu, Y. Liu, C. Ma, S. Wei, Z. He, J. Li, and E. Y. K. Ng, "An open access database for evaluating the algorithms of electrocardiogram rhythm and morphology abnormality detection," Journal of Medical Imaging and Health Informatics, vol. 8, pp. 1368-1373, 2018.
[4] H. Gao, C. Liu, X. Wang, L. Zhao, Q. Shen, E. Y. K. Ng, and J. Li, "An Open-Access ECG Database for Algorithm Evaluation of QRS Detection and Heart Rate Estimation," Journal of Medical Imaging and Health Informatics, vol. 9, pp. 1853-1858, 2019.
[5] J. Oster, J. Behar, O. Sayadi, S. Nemati, A. E. Johnson, and G. D. Clifford, "Semisupervised ECG ventricular beat classification with novelty detection based on switching Kalman filters," IEEE Transactions on Biomedical Engineering, vol. 62, pp. 2125-2134, 2015.
[6] A. L. Goldberger, L. A. Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C. Peng, and H. E. Stanley, "PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals," Circulation, vol. 101, pp. e215-e220, 2000.
[7] F. Liu, C. Liu, X. Jiang, Z. Zhang, Y. Zhang, J. Li, and S. Wei, "Performance analysis of ten common QRS detectors on different ECG application cases," Journal of Healthcare Engineering, vol. 2018, pp. 9050812(1)-9050812(8), 2018.
[8] ANSI/AAMI EC57, "1998 / (R) 2008-Testing and reporting performance results of cardiac rhythm and ST segment measurement algorithms", Arlington, VA, USA, 2008.