Searching for Premature Ventricular Contraction and Supraventricular Premature Beat from Long-term ECGs: The 3rd China Physiological Signal Challenge 2020

---

If you use the Challenge data for paper publication, please cite this paper for Challenge data description:
Z.P. Cai, C.Y. Liu*, H.X. Gao, X.Y. Wang, L.N. Zhao, Q. Shen, E. Y. K. Ng, and Jianqing Li*. An Open-Access Long-Term Wearable ECG Database for Premature Ventricular Contractions and Supraventricular Premature Beat Detection. Journal of Medical Imaging and Health Informatics, 2020, 10(11): 2663-2667.

News

The final challenge scores of CPSC 2020 are shown as below, with the final code available as open-source data.

Code No.	Institution/Affiliation	Team Members	Score PVC	Score SPB
CPSC1077	Shinall Technology	Min Chen, Kui Dong	41479	92947
CPSC1091	University of Shanghai for Science and Technology	Wenjie Cai, Jingying Yang, Jianjian Cao, Xuan Wang	55706	120942
CPSC1093	1. Soochow University; 2. Suzhou Institute of Biomedical Engineering and Technology Chinese Academy of Sciences	Lirong Wang1, Lishen Qiu2, Wenqiang Cai1, Wenliang Zhu2, Jie Yu1, Wanyue Li1, Duoduo Wang1, Huimin Zhang1	95900	111523
CPSC1082	Beijing University of Technology	Minggang Shao, Zhuhuang Zhou, Shuicai Wu	97913	95348
CPSC1089	Chengdu Spaceon Electronics CO., LTD.	Shan Yang, Chunli Wang, Heng Xiang, Qingda Kong	142228	117942
CPSC1104	Tsinghua University	Hao Wen	143484	99824
CPSC1085	Taiwan AI Academy; Academia Sinica; National Taiwan University	Tsai-Min Chen	144966	153040
CPSC1098	Northeastern University	Yan Li, Yuxiang Li, Haixu Yang, Jihong Liu	151735	215664
CPSC1092	Harbin Institute of Technology	Yang Liu, Runnan He	166215	160474
CPSC1081	Institute of Semiconductors, Chinese Academy of Sciences; University of Chinese Academy of Sciences	Yibo Yin; Sitao Zhang	168578	195467
CPSC1088	East China Jiaotong University	Feng Mei, Qian Hu, Lingfeng Liu	362348	120410

Notes:
[2020-10-16] New Sample Python entry has been updated to avoid repeated counting false posibility(FP).

[2020-8-31] Sample Python entry has been updated for issues in FP statistics.

[2020-8-18] In response to the training set label problem raised by some contestants, we confirmed and updated all the labels, please download the new Training Set for your code preparation.

[2020-8-12] It is necessary to to provide detailed descriptions and version numbers of dependent third-party libraries in the form of requirements.txt. It is recommended to apply for tensorflow>=1.13.1, or tensorflow>=2.0, keras>=2.2.4,pytorch.

[2020-8-12] Sample Python entry has been updated for cross-system issues.

The Score List has been indicated as below, more will be updated soon…

Code No.	Institution/Affiliation	Team Members	ScorePVC	ScoreSPB
CPSC841	Shinall Technology	Min Chen, Kui Dong	12246	36640
CPSC775	Beijing University of Technology	Minggang Shao, Zhuhuang Zhou, Shuicai Wu	15062	77440
CPSC1054	University of Shanghai for Science and Technology	Wenjie Cai, Jingying Yang, Jianjian Cao, Xuan Wang	21285	50941
CPSC982	Harbin Institute of Technology	Yang Liu, Runnan He	23654	101026
CPSC769	Chengdu Spaceon Electronics CO., LTD.	Shan Yang, Chunli Wang, Heng Xiang, Qingda Kong	33597	73773
CPSC1051	Northeastern University	Yan Li, Yuxiang Li, Haixu Yang, Jihong Liu	40152	124095
CPSC1033	1. Soochow University; 2. Suzhou Institute of Biomedical Engineering and Technology Chinese Academy of Sciences	Lirong Wang1, Lishen Qiu2, Wenqiang Cai1, Wenliang Zhu2, Jie Yu1, Wanyue Li1, Duoduo Wang1, Huimin Zhang1	45785	74590
CPSC897	Shenzhen Soonview Technology Co., Ltd.	Longbo Zhang, Chi Wang	62955	79895
CPSC917	Taiwan AI Academy; Academia Sinica; National Taiwan University	Tsai-Min Chen	63192	80772
CPSC950	Shanghai National Group Health Technology Co., Ltd	Jie Yang	63187	80772
CPSC664	Institute of Scientific Instruments of the CAS Brno	Adam Ivora	76290	87382
CPSC758	Xi'an Jiaotong University; The University of Manchester; University of Oxford	Jingyao Wu, Zhibin Zhao, Ruqiang Yan; David C. Wong; Nicola K Dinsdale	89310	77900
CPSC829	Tsinghua University	Hao Wen	94962	53300
CPSC1027	Institute of Semiconductors, Chinese Academy of Sciences; University of Chinese Academy of Sciences	Yibo Yin, Sitao zhang	88257	98543
CPSC933	East China Jiaotong University	Feng Mei, Qian Hu, Lingfeng Liu	141159	165949
CPSC921	Northeastern University	Haixu Yang, Yan Li, Jihong Liu	169794	253054
CPSC731	Nalong Technology	Xi Li	180725	78890
CPSC722	Soochow University	Yifan Li	227492	100140
CPSC748	Qatar University; Izmir University of Economics	Muhammad Uzair Zahid, Ozer Can Devecioglu, Turker Ince, Mustafa Serkan Kiranyaz	227650	98740
CPSC802	Zhejiang University	Shao	227650	100185
CPSC794	Universidad Nacional Autónoma de México	Jonathan Roberto Torres Castillo, Karen Patricia Gaitan de lo Rios, Miguel Angel Padilla Castañeda	227650	100185
CPSC727	Lenovo Group Ltd.	Ping Xu	227650	100185
CPSC953	Sichuan University	Lei Xiong, Leiliang Qin	227650	100185
CPSC766	University of Electronic Science and Technology of China	Jin Qi, Sunfeng Luo, Xiaofang He	227650	100185
CPSC789	Ludong University	Shengqiang Liu, Laiguo Li, Qingran Dong, Shuo Zhu, Lijun Guo, Wenxing Shi, Qian Li, Xiaochen Wei, Jing Li	227650	100634
CPSC797	Ludong University	Aowei Liu, Zixuan Liu	227651	100183

Frequently Asked Questions (FAQs)

Q1: How can we register team information to join the challenge?
R1: We suggest that the team leader, or one of the team representatives registers an account on behalf of the team in the system. Please DO NOT register multiple accounts from one team.

Q2: How should I submit the challenge entries with the provided sample entry ‘CPSC2020_challenge’?
R2: You can add the signal processing codes in the ‘CPSC2020_challenge.m', and submit the updated version of 'CPSC2020_challenge.m'. At this moment, please focus on the methods you need to develop, and submit the ‘CPSC2020_challenge.m’ with developed codes. Test data and training data are exactly in the same format.

Q3: How should I name the main function?
R3: Please name it CPSC2020_challenge.

Q4: If we use the python environment, can the test system support commonly used packages, such as keras, wfdb, numpy?
R4: Yes, It can.

Q5: How can I contact the organizing committee?
R5: Please join the Wechat Group or email to chinachallenge_cpsc@icbeb.org.

Introduction

Abnormality of cardiac conduction system can induce arrhythmia. Abnormal heart rhythm can lead to other cardiac diseases and complications, and can be life-threatening [1]. There are various types of arrhythmias and each type is associated with a pattern, and as such, it is possible to be identified. Arrhythmias can be classified into two major categories. The first category consists of arrhythmias formed by a single irregular heartbeat in electrocardiogram (ECG), herein called morphological arrhythmia, while another category consists of arrhythmias formed by a set of irregular heartbeats in ECG, herein called rhythmic arrhythmias [2]. Dynamic electrocardiogram (DCG), like ECG Holter, provides an important way to monitor the incidences of arrhythmias in daily life, facilitating the doctors to check a total number and distribution of arrhythmias in a long time and thus to provide the required therapy to prevent further problems.
The 3rd China Physiological Signal Challenge 2020 (CPSC 2020) aims to encourage the development of algorithms for searching for premature ventricular contraction (PVC) and supraventricular premature beat (SPB) from 24-hour dynamic single-lead ECG recordings usually with low signal quality and/or abnormal rhythm waveforms. Similar the previous works and efforts of the CPSC 2018 [3] and CPSC 2019 [4], accurate locating of abnormal heartbeats is another critical issue put forward here for further discussion.
ECG signal provides an important role in non-invasively monitoring and clinical diagnosis for cardiovascular disease (CVD). Arrhythmia detection is one of the ultimate goals of routine ECG monitoring, and PVC and SPB are the two most common arrhythmias. Increase in these beats may be a precursor to stroke or sudden cardiac death [5]. Although their detection methods have been severely tracked throughout the last several decades, accurate and robust detections are still challenging in noisy or low-signal quality environment, especially for daily monitored ECG waveforms. It is true that many of the developed PVC and SPB detection algorithms can achieve high accuracy (over 96% in sensitivity and positive predictivity) when tested over the standard ECG databases such as the MIT-BIH Arrhythmia Database or AHA Database [6]. However, these algorithms may fail when used in the noisy environment. Especially, even the basic QRS detection can be invalid in the low signal quality ECG analysis [7]. A recent study confirmed that none of the common QRS detection algorithms can obtain 80% detection accuracy when tested in a dynamic noisy ECG database. In this year’s challenge, we provide a new ECG database containing long-term noisy ECG recordings from clinical arrhythmia patients, to encourage the participants to develop more efficient and robust algorithms for PVC and SPB detection.

Challenge Data

Training data consists of 10 single-lead ECG recordings collected from arrhythmia patients, each of the recording last for about 24 hours (shown in Table 1). Table 1 also indicates the patient if he/she is an atrial fibrillation (AF) patient. Test set contains similar ECG recordings, which is unavailable to public and will remain private for the purpose of scoring for the duration of Challenge and for some period afterwards. All data were collected by a unified wearable ECG device with a sampling frequency of 400 Hz, and provided in MATLAB format (each including three *.mat file: one is ECG data and another two are the corresponding PVC and SPB annotation files, respectively).
Table 1: Detailed information of training data.

Recordings	AF patient ?	Length (h)	# N beats	# V beats	# S beats	# Total beats
A01	No	25.89	109,062	0	24	109,086
A02	Yes	22.83	98,936	4,554	0	103,490
A03	Yes	24.70	137,249	382	0	137,631
A04	No	24.51	77,812	19,024	3,466	100,302
A05	No	23.57	94,614	1	25	94,640
A06	No	24.59	77,621	0	6	77,627
A07	No	23.11	73,325	15,150	3,481	91,956
A08	Yes	25.46	115,518	2,793	0	118,311
A09	No	25.84	88,229	2	1,462	89,693
A10	No	23.64	72,821	169	9,071	82,061

Although PVC and SPB detection algorithms are widely studied for many years, accurate detection is still challenging in this Challenge due to the amplitude variation, morphological variation, as well as noises in ECG data. Figure 1 shows examples of ECG waveforms in training data.

Figure 1. Examples of demonstrated ECG waveforms. Black circles denote the reference locations of SPB and red ones denote the reference locations of PVC. The challenges for accurate PVC and SPB detection are from: A) amplitude variability and multi-source PVC; B) noise and other abnormal interference (such as AF) and C) electrode sliding interference.

Challenge Events and Scoring

CPSC 2020 is comprised of two events related to scoring: PVC detection and SPB detection. PVC and SPB annotations in the training and test sets are labeled and initially confirmed by cardiologists and trained volunteers. Score is calculated according to the following rules.
Event 1: PVC detection
In this event, the goal is to generate a set of PVC annotations for each recording that can matches the reference PVC annotations. For each reference PVC annotation, a matched PVC annotation should lie in 150 ms duration centered by the reference PVC annotation [8]. Noted that the reference PVC annotations appear in the first and last 0.2 seconds are ignored. Detected PVC should be within 150 ms from the reference ones. The scoring rules are:
    •   a false positive ( FP) detection deduct 1 point.
    •   a false negative ( FN) detection deduct 5 points, since from a clinical perspective, missed diagnosis is more serious than misdiagnosis, thus we penalize FN detection.
        The final score for Event 1 ( PVC _err) is the sum of all deducted points.
Event 2: SPB detection
In this event, the goal is to generate a set of SPB annotations for each recording that can matches the reference SPB annotations. For each reference SPB annotation, a matched SPB annotation should lie in 150 ms duration centered by the reference SPB annotation [8]. Noted that the reference SPB annotations appear in the first and last 0.2 seconds are also ignored. Detected SPB should be within 150 ms from the reference ones. The scoring rules are:
    •   a false positive ( FP) detection deduct 1 point.
    •   a false negative ( FN) detection deduct 5 points, since from a clinical perspective, missed diagnosis is more serious than misdiagnosis, thus we penalize FN detection.
        The final score for Event 2 ( SPB _err) is the sum of all deducted points.

Quick Start

1. Download the training set[2020-8-18] and the sample entry ( Sample MATLAB entry/ Sample Python entry[2020-8-12] ).
2. Create a free account and join the CPSC Challenge 2020 project.
3. Develop your entry by editing the existing files:
    •   You are allowed modify the sample entry source code file CPSC2020_challenge.m with your changes and improvements. But, never alter any word in CPSC2020_score.m to avoid a wrong result.
    •   Modify the AUTHORS.txt file to include the names of all the team members.
    •   Run your modified source code file on all the records in the training set by executing the script CPSC2020_score.m. This will also build a new version of entry.zip.
    •   Optional: Include a file named DRYRUN in the top directory of your entry (where the AUTHORS.txt file is located) if you do not wish your entry to be scored and counted against your limit. This is useful in cases where you wish to make sure that the changes made do not result in any error.
4. Please note: we cannot guarantee the 100% accuracy for the reference PVC/SPB annotations. If you find that the reference is questioned (error), please give us a feedback .txt file using the following format to us:
File name: "data label name - event type (S / V) - sampling point position - expected result" (example: A07-S-76214-N.txt). Please keep the ‘.txt’ content empty.
We will further verify this with the cardiologists, and update the reference annotations in the website periodically.

Participating in the challenge

To participate in the challenge, you need to submit a software written by MATLAB or Python that is able to run on the test set and output the final scoring result without user interaction in our test environment. One sample entry CPSC2020_challenge.m is available to help you get started. And a scoring function CPSC2020_score.m is provided for competitors testing algorithms and build a new version of entry.zip. The team must submit the first challenge open-source entry before July 1, 2020. The deadline for submitting the official challenge entries is October 25, 2020.

Awards and Rules

The winner will be selected on the basis of the obtained final PVC _err and SPB _err on the hidden test data. The first three for each Event challenging will receive certificates and generous bonuses:
    •    First prize: Certificate plus bonus of RMB 15,000
    •   Second prize: Certificate plus bonus of RMB 10,000
    •   Third prize: Certificate plus bonus of RMB 5,000
We welcome all the individual or research group around the world to attend the challenge. To be eligible for the open-source award, please do the following (see table 2):
    (1) Register the team before June 1, 2020.
    (2) Submit at least one open-source entry that can be scored before July 1, 2020.
    (3) Attend ICBEB 2020 (November 15-18, 2020) and present your work there.
All deadlines occur at 11:59pm GMT+8 (UTC) on the dates mentioned below. If you do not know the difference between GMT and your local time, find out what it is before the deadline!
Table 2: Challenge deadlines.

	Start	Entry limit	End
Registration	1 February	At least once	30 June
First Stage	1 July	10	31 September
Second Stage	1 October	3	25 October
Final Stage	26 October	Final entry	1 November

In the interest of fairness to all participants, any entries received later than deadline of that stage will not be accepted or scored. Code submitted more than the limit number of times will not be scored.
Please do not submit analysis of this year’s Challenge data to other Conferences or Journals until after ICBEB 2020 has taken place, so the competitors are able to discuss the results in a single forum.

Important Dates
15 February, 2020 -- Challenge open
June 1, 2020 -- Deadline for team registration to join the challenge
July 1, 2020 -- Deadline for submitting the first challenge open-source entry
October 25, 2020 -- Deadline for submitting the challenge entries
November 1, 2020 -- Confirm for submitting the best entry for final scoring
November 15-18, 2020 – Announcement of the winners of CPSC 2020 in ICBEB 2020

Any questions or problems about the Challenge, please feel free to contact chinachallenge_cpsc@icbeb.org

Challenge Chair:
Prof. Chengyu Liu, Southeast University, China
Challenge Committee:
Dr. Zhipeng Cai, Southeast University, China
Dr. Hongxiang Gao, Southeast University, China
Dr. Xingyao Wang, Southeast University, China
Dr. Qin Shen, Nanjing Medical University, China
Dr. Xiangwei Ding, Nanjing Medical University, China
International Advisory Chair:
Prof. Gari D. Clifford, Emory University & Georgia Institute of Technology, USA
International Advisory Co-chairs:
Prof. Aiguo Song, Southeast University, China
Prof. Jianqing Li, Nanjing Medical University, China
Prof. Zuhong Lu, Southeast University, China
Prof. Yi Peng, Peking Union Medical College, China
Mr. Yingjia Yao, Lenovo Group, China
International Advisory Committee:
Prof. Eddie Ng Yin Kwee, Nanyang Technological University, Singapore
Prof. Kang-Ping Lin, Chung Yuan Christian University, Chinese Taipei
Prof. Feng Wang, Southeast University, China
Prof. Zhengtao Cao, Air Force Medical Center, China
Prof. Xianzheng Sha, China Medical University, China
Prof. Xingming Guo, Chongqing University, China
Prof. Shoushui Wei, Shandong University, China
Prof. Zhi Tao, Suzhou University, China
Dr. Alistair Johnson, MIT, USA

Hosted by:
School of Instrument Science and Engineering, Southeast University, China
The State Key Laboratory of Bioelectronics, Southeast University, China
School of Biomedical Engineering and Information, Nanjing Medical University, China

Supported by:
ICBEB Organizing Committee
Health Engineering Committee of Chinese Society of Biomedical Engineering
Biomedical Sensor Committee of Chinese society of Biomedical Engineering
Youth Working Committee of Chinese Society of Biomedical Engineering
Jiangsu Instrument and Control Society

Awards sponsored by:
Lenovo Group

Reference
[1] S. L. Oh, E. Y. Ng, R. San Tan, and U. R. Acharya, "Automated diagnosis of arrhythmia using combination of CNN and LSTM techniques with variable length heart beats," Computers in biology and medicine, vol. 102, pp. 278-287, 2018.
[2] E. J. D. S. Luz, W. R. Schwartz, G. Cámara-Chávez, and D. Menotti, "ECG-based heartbeat classification for arrhythmia detection: A survey," Computer methods and programs in biomedicine, vol. 127, pp. 144-164, 2016.
[3] F. Liu, C. Liu, L. Zhao, X. Zhang, X. Wu, X. Xu, Y. Liu, C. Ma, S. Wei, Z. He, J. Li, and E. Y. K. Ng, "An open access database for evaluating the algorithms of electrocardiogram rhythm and morphology abnormality detection," Journal of Medical Imaging and Health Informatics, vol. 8, pp. 1368-1373, 2018.
[4] H. Gao, C. Liu, X. Wang, L. Zhao, Q. Shen, E. Y. K. Ng, and J. Li, "An Open-Access ECG Database for Algorithm Evaluation of QRS Detection and Heart Rate Estimation," Journal of Medical Imaging and Health Informatics, vol. 9, pp. 1853-1858, 2019.
[5] J. Oster, J. Behar, O. Sayadi, S. Nemati, A. E. Johnson, and G. D. Clifford, "Semisupervised ECG ventricular beat classification with novelty detection based on switching Kalman filters," IEEE Transactions on Biomedical Engineering, vol. 62, pp. 2125-2134, 2015.
[6] A. L. Goldberger, L. A. Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C. Peng, and H. E. Stanley, "PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals," Circulation, vol. 101, pp. e215-e220, 2000.
[7] F. Liu, C. Liu, X. Jiang, Z. Zhang, Y. Zhang, J. Li, and S. Wei, "Performance analysis of ten common QRS detectors on different ECG application cases," Journal of Healthcare Engineering, vol. 2018, pp. 9050812(1)-9050812(8), 2018.
[8] ANSI/AAMI EC57, "1998 / (R) 2008-Testing and reporting performance results of cardiac rhythm and ST segment measurement algorithms", Arlington, VA, USA, 2008.