Rapid artificial intelligence solutions in a pandemic—The COVID-19-20 Lung CT Lesion Segmentation Challenge

[en] Artificial intelligence (AI) methods for the automatic detection and quantification of COVID-19 lesions in chest computed tomography (CT) might play an important role in the monitoring and management of the disease. We organized an international challenge and competition for the development and comparison of AI algorithms for this task, which we supported with public data and state-of-the-art benchmark methods. Board Certified Radiologists annotated 295 public images from two sources (A and B) for algorithms training (n=199, source A), validation (n=50, source A) and testing (n=23, source A; n=23, source B). There were 1,096 registered teams of which 225 and 98 completed the validation and testing phases, respectively. The challenge showed that AI models could be rapidly designed by diverse teams with the potential to measure disease or facilitate timely and patient-specific interventions. This paper provides an overview and the major outcomes of the COVID-19 Lung CT Lesion Segmentation Challenge - 2020.

Disciplines :

Engineering, computing & technology: Multidisciplinary, general & others

Author, co-author :

Roth, Holger R.

Xu, Ziyue

Diez, Carlos Tor

Jacob, Ramon Sanchez

Zember, Jonathan

Molto, Jose

Li, Wenqi

Xu, Sheng

Turkbey, Baris

Turkbey, Evrim

More authors (26 more)

External co-authors :

yes

Language :

English

Title :

Rapid artificial intelligence solutions in a pandemic—The COVID-19-20 Lung CT Lesion Segmentation Challenge

Publication date :

2022

Journal title :

Medical Image Analysis

ISSN :

1361-8415

Pages :

102605

Peer reviewed :

Peer reviewed

Focus Area :

Computational Sciences

Additional URL :

https://www.sciencedirect.com/science/article/pii/S1361841522002353

Available on ORBilu :

since 12 September 2022

Statistics

Number of views

57 (5 by Unilu)

Number of downloads

0 (0 by Unilu)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

OpenAlex citations

WoS citations^™

Bibliography

Andrearczyk, Vincent, Oreiller, Valentin, Depeursinge, Adrien, Head and neck tumor segmentation: First challenge, HECKTOR 2020, held in conjunction with MICCAI 2020, lima, peru, october 4, 2020, proceedings. 2021, Springer Nature.
Bai, Harrison X, Wang, Robin, Xiong, Zeng, Hsieh, Ben, Chang, Ken, Halsey, Kasey, Tran, Thi My Linh, Choi, Ji Whae, Wang, Dong-Cui, Shi, Lin-Bo, et al. Artificial intelligence augmentation of radiologist performance in distinguishing COVID-19 from pneumonia of other origin at chest CT. Radiology 296:3 (2020), E156–E165.
Bao, Cuiping, Liu, Xuehuan, Zhang, Han, Li, Yiming, Liu, Jun, Coronavirus disease 2019 (COVID-19) CT findings: a systematic review and meta-analysis. J. Am. College Radiol. 17:6 (2020), 701–709.
Bernheim, Adam, Mei, Xueyan, Huang, Mingqian, Yang, Yang, Fayad, Zahi A, Zhang, Ning, Diao, Kaiyue, Lin, Bin, Zhu, Xiqi, Li, Kunwei, et al. Chest CT findings in coronavirus disease-19 (COVID-19): relationship to duration of infection. Radiology, 2020, 200463.
Cao, Yinghao, Liu, Xiaoling, Xiong, Lijuan, Cai, Kailin, Imaging and clinical features of patients with 2019 novel coronavirus SARS-CoV-2: a systematic review and meta-analysis. J. Med. Virol. 92:9 (2020), 1449–1459.
Çiçek, Özgün, Abdulkadir, Ahmed, Lienkamp, Soeren S, Brox, Thomas, Ronneberger, Olaf, 3D U-net: learning dense volumetric segmentation from sparse annotation. International Conference on Medical Image Computing and Computer-Assisted Intervention, 2016, Springer, 424–432.
Clark, Kenneth, Vendt, Bruce, Smith, Kirk, Freymann, John, Kirby, Justin, Koppel, Paul, Moore, Stephen, Phillips, Stanley, Maffitt, David, Pringle, Michael, et al. The cancer imaging archive (TCIA): maintaining and operating a public information repository. J. Digital Imag. 26:6 (2013), 1045–1057.
Dayan, Ittai, Roth, Holger R, Zhong, Aoxiao, Harouni, Ahmed, Gentili, Amilcare, Abidin, Anas Z, Liu, Andrew, Costa, Anthony Beardsworth, Wood, Bradford J, Tsai, Chien-Sung, et al. Federated learning for predicting clinical outcomes in patients with COVID-19. Nat. Med., 2021, 1–9.
Desai, Shivang, Baghal, Ahmad, Wongsurawat, Thidathip, Jenjaroenpun, Piroon, Powell, Thomas, Al-Shukri, Shaymaa, Gates, Kim, Farmer, Phillip, Rutherford, Michael, Blake, Geri, et al. Chest imaging representing a COVID-19 positive rural US population. Sci. Data 7:1 (2020), 1–6.
Drozdzal, Michal, Vorontsov, Eugene, Chartrand, Gabriel, Kadoury, Samuel, Pal, Chris, The importance of skip connections in biomedical image segmentation. Deep Learning and Data Labeling for Medical Applications, 2016, Springer, 179–187.
Eugster, Manuel J.A., Hothorn, Torsten, Leisch, Friedrich, Exploratory and inferential analysis of benchmark experiments. Tech. Rep., No.30, 2008.
Fan, Deng-Ping, Zhou, Tao, Ji, Ge-Peng, Zhou, Yi, Chen, Geng, Fu, Huazhu, Shen, Jianbing, Shao, Ling, Inf-net: Automatic covid-19 lung infection segmentation from ct images. IEEE Trans. Med. Imaging 39:8 (2020), 2626–2637.
Haque, Intisar Rizwan I., Neubert, Jeremiah, Deep learning approaches to biomedical image segmentation. Inf. Med. Unlocked, 18, 2020, 100297.
Harmon, Stephanie A, Sanford, Thomas H, Xu, Sheng, Turkbey, Evrim B, Roth, Holger, Xu, Ziyue, Yang, Dong, Myronenko, Andriy, Anderson, Victoria, Amalou, Amel, et al. Artificial intelligence for the detection of COVID-19 pneumonia on chest CT using multinational datasets. Nature Commun. 11:1 (2020), 1–7.
Hopkins, Johns, Home - johns hopkins coronavirus resource center. 2021 https://coronavirus.jhu.edu/. (Accessed 17 May 2021).
Huang, Lu, Han, Rui, Ai, Tao, Yu, Pengxin, Kang, Han, Tao, Qian, Xia, Liming, Serial quantitative chest CT assessment of COVID-19: a deep learning approach. Radiol. Cardiothoracic Imaging, 2(2), 2020, e200075.
Ippolito, Davide, Ragusi, Maria, Gandola, Davide, Maino, Cesare, Pecorelli, Anna, Terrani, Simone, Peroni, Marta, Giandola, Teresa, Porta, Marco, Franzesi, Cammillo Talei, et al. Computed tomography semi-automated lung volume quantification in SARS-CoV-2-related pneumonia. Euro. Radiol. 31:5 (2021), 2726–2736.
Isensee, Fabian, Jaeger, Paul F, Kohl, Simon AA, Petersen, Jens, Maier-Hein, Klaus H, NnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods 18:2 (2021), 203–211.
Kang, Hengyuan, Xia, Liming, Yan, Fuhua, Wan, Zhibin, Shi, Feng, Yuan, Huan, Jiang, Huiting, Wu, Dijia, Sui, He, Zhang, Changqing, et al. Diagnosis of coronavirus disease 2019 (COVID-19) with structured latent multi-view representation learning. IEEE Trans. Med. Imaging 39:8 (2020), 2606–2614.
Larici, Anna Rita, Cicchetti, Giuseppe, Marano, Riccardo, Merlino, Biagio, Elia, Lorenzo, Calandriello, Lucio, Del Ciello, Annemilia, Farchione, Alessandra, Savino, Giancarlo, Infante, Amato, et al. Multimodality imaging of COVID-19 pneumonia: from diagnosis to follow-up. A comprehensive review. Eur. J. Radiol., 2020, 109217.
LeCun, Yann, Bengio, Yoshua, Hinton, Geoffrey, Deep learning. Nature 521:7553 (2015), 436–444.
Li, Lin, Qin, Lixin, Xu, Zeguo, Yin, Youbing, Wang, Xin, Kong, Bin, Bai, Junjie, Lu, Yi, Fang, Zhenghan, Song, Qi, et al. Using artificial intelligence to detect COVID-19 and community-acquired pneumonia based on pulmonary CT: evaluation of the diagnostic accuracy. Radiology 296:2 (2020), E65–E71.
Lin, Tsung-Yi, Goyal, Priya, Girshick, Ross, He, Kaiming, Dollár, Piotr, Focal loss for dense object detection. Proceedings of the IEEE International Conference on Computer Vision, 2017, 2980–2988.
Liu, Hanxiao, Simonyan, Karen, Yang, Yiming, Darts: Differentiable architecture search. 2018 arXiv preprint arXiv:1806.09055.
Long, Jonathan, Shelhamer, Evan, Darrell, Trevor, 2015. Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 3431–3440.
Lyu, Siwei, Fan, Yanbo, Ying, Yiming, Hu, Bao-Gang, Average top-k aggregate loss for supervised learning. IEEE Trans. Pattern Anal. Mach. Intell., 2020.
Maier-Hein, Lena, Eisenmann, Matthias, Reinke, Annika, Onogur, Sinan, Stankovic, Marko, Scholz, Patrick, Arbel, Tal, Bogunovic, Hrvoje, Bradley, Andrew P, Carass, Aaron, et al. Why rankings of biomedical image analysis competitions should be interpreted with care. Nature Commun. 9:1 (2018), 1–13.
Mei, Xueyan, Lee, Hao-Chih, Diao, Kai-yue, Huang, Mingqian, Lin, Bin, Liu, Chenyu, Xie, Zongyu, Ma, Yixuan, Robson, Philip M, Chung, Michael, et al. Artificial intelligence–enabled rapid diagnosis of patients with COVID-19. Nat. Med. 26:8 (2020), 1224–1228.
Merkus, Peter J.F.M., Klein, Willemijn M., The value of chest CT as a COVID-19 screening tool in children. Eur. Respir. J., 55(6), 2020.
Milletari, Fausto, Navab, Nassir, Ahmadi, Seyed-Ahmad, V-net: Fully convolutional neural networks for volumetric medical image segmentation. 2016 Fourth International Conference on 3D Vision (3DV), 2016, IEEE, 565–571.
Nino, Gustavo, Zember, Jonathan, Sanchez-Jacob, Ramon, Gutierrez, Maria J, Sharma, Karun, Linguraru, Marius George, Pediatric lung imaging features of COVID-19: a systematic review and meta-analysis. Pediatr. Pulmonol. 56:1 (2021), 252–263.
Ojha, Vineeta, Mani, Avinash, Pandey, Niraj Nirmal, Sharma, Sanjiv, Kumar, Sanjeev, CT in coronavirus disease 2019 (COVID-19): a systematic review of chest CT findings in 4410 adult patients. Euro. Radiol. 30 (2020), 6129–6138.
Oulefki, Adel, Agaian, Sos, Trongtirakul, Thaweesak, Laouar, Azzeddine Kassah, Automatic COVID-19 lung infected region segmentation and measurement using CT-scans images. Pattern Recognit., 114, 2021, 107747.
Patrick, Sanjana, Birur, N Praveen, Gurushanth, Keerthi, Raghavan, A Shubhasini, Gurudath, Shubha, et al. Comparison of gray values of cone-beam computed tomography with hounsfield units of multislice computed tomography: an in vitro study. Indian J. Dental Res., 28(1), 2017, 66.
Project MONAI, Project MONAI. 2021 https://monai.io/. (Accessed September 22, 2022).
Roberts, Michael, Driggs, Derek, Thorpe, Matthew, Gilbey, Julian, Yeung, Michael, Ursprung, Stephan, Aviles-Rivero, Angelica I, Etmann, Christian, McCague, Cathal, Beer, Lucian, et al. Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans. Nat. Mach. Intell. 3:3 (2021), 199–217.
Ronneberger, Olaf, Fischer, Philipp, Brox, Thomas, U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention, 2015, Springer, 234–241.
Rubin, Geoffrey D, Ryerson, Christopher J, Haramati, Linda B, Sverzellati, Nicola, Kanne, Jeffrey P, Raoof, Suhail, Schluger, Neil W, Volpi, Annalisa, Yim, Jae-Joon, Martin, Ian BK, et al. The role of chest imaging in patient management during the COVID-19 pandemic: a multinational consensus statement from the Fleischner Society. Radiology 296:1 (2020), 172–180.
Shan, Fei, Gao, Yaozong, Wang, Jun, Shi, Weiya, Shi, Nannan, Han, Miaofei, Xue, Zhong, Shen, Dinggang, Shi, Yuxin, Abnormal lung quantification in chest CT images of COVID-19 patients with deep learning and its application to severity prediction. Med. Phys. 48:4 (2021), 1633–1645.
Sun, Dong, Li, Xiang, Guo, Dajing, Wu, Lan, Chen, Ting, Fang, Zheng, Chen, Linli, Zeng, Wenbing, Yang, Ran, CT quantitative analysis and its relationship with clinical features for assessing the severity of patients with COVID-19. Korean J. Radiol., 21(7), 2020, 859.
Tajbakhsh, Nima, Jeyaseelan, Laura, Li, Qian, Chiang, Jeffrey N, Wu, Zhihao, Ding, Xiaowei, Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation. Med. Image Anal., 63, 2020, 101693.
Wan, Shang, Li, Mingqi, Ye, Zheng, Yang, Caiwei, Cai, Qian, Duan, Shaofeng, Song, Bin, CT manifestations and clinical characteristics of 1115 patients with coronavirus disease 2019 (COVID-19): a systematic review and meta-analysis. Acad. Radiol. 27:7 (2020), 910–921.
Wang, Jun, Bao, Yiming, Wen, Yaofeng, Lu, Hongbing, Luo, Hu, Xiang, Yunfei, Li, Xiaoming, Liu, Chen, Qian, Dahong, Prior-attention residual learning for more discriminative COVID-19 screening in CT images. IEEE Trans. Med. Imaging 39:8 (2020), 2572–2583.
Wang, Guotai, Liu, Xinglong, Li, Chaoping, Xu, Zhiyong, Ruan, Jiugen, Zhu, Haifeng, Meng, Tao, Li, Kang, Huang, Ning, Zhang, Shaoting, A noise-robust framework for automatic segmentation of COVID-19 pneumonia lesions from CT images. IEEE Trans. Med. Imaging 39:8 (2020), 2653–2663.
Wiesenfarth, Manuel, Reinke, Annika, Landman, Bennett A, Eisenmann, Matthias, Saiz, Laura Aguilera, Cardoso, M Jorge, Maier-Hein, Lena, Kopp-Schneider, Annette, Methods and open-source toolkit for analyzing and visualizing challenge results. Sci. Rep. 11:1 (2021), 1–15.
Wynants, Laure, Van Calster, Ben, Collins, Gary S, Riley, Richard D, Heinze, Georg, Schuit, Ewoud, Bonten, Marc MJ, Dahly, Darren L, Damen, Johanna A, Debray, Thomas PA, et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. Bmj, 369, 2020.
Yang, Dong, Roth, Holger, Xu, Ziyue, Milletari, Fausto, Zhang, Ling, Xu, Daguang, Searching learning strategy with reinforcement learning for 3d medical image segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention, 2019, Springer, 3–11.
Yang, Dong, Xu, Ziyue, Li, Wenqi, Myronenko, Andriy, Roth, Holger R, Harmon, Stephanie, Xu, Sheng, Turkbey, Baris, Turkbey, Evrim, Wang, Xiaosong, et al. Federated semi-supervised learning for COVID region segmentation in chest CT using multi-national data from China, Italy, Japan. Med. Image Anal., 70, 2021, 101992.
Yu, Qihang, Yang, Dong, Roth, Holger, Bai, Yutong, Zhang, Yixiao, Yuille, Alan L, Xu, Daguang, 2020. C2fnas: Coarse-to-fine neural architecture search for 3d medical image segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4126–4135.
Yushkevich, Paul A, Piven, Joseph, Hazlett, Heather Cody, Smith, Rachel Gimpel, Ho, Sean, Gee, James C, Gerig, Guido, User-guided 3D active contour segmentation of anatomical structures: significantly improved efficiency and reliability. Neuroimage 31:3 (2006), 1116–1128.
Zang, Si-Tian, Han, Xu, Cui, Qi, Chang, Qing, Wu, Qi-Jun, Zhao, Yu-Hong, Imaging characteristics of coronavirus disease 2019 (COVID-19) in pediatric cases: a systematic review and meta-analysis. Transl. Pediatrics, 10(1), 2021, 1.
Zhang, Kang, Liu, Xiaohong, Shen, Jun, Li, Zhihuan, Sang, Ye, Wu, Xingwang, Zha, Yunfei, Liang, Wenhua, Wang, Chengdi, Wang, Ke, et al. Clinically applicable AI system for accurate diagnosis, quantitative measurements, and prognosis of COVID-19 pneumonia using computed tomography. Cell 181:6 (2020), 1423–1433.
Zhou, Tongxue, Canu, Stéphane, Ruan, Su, Automatic COVID-19 CT segmentation using U-net integrated spatial and channel attention mechanism. Int. J. Imaging Syst. Technol. 31:1 (2021), 16–27.
Zhu, Zhuotun, Liu, Chenxi, Yang, Dong, Yuille, Alan, Xu, Daguang, V-nas: Neural architecture search for volumetric medical image segmentation. 2019 International Conference on 3D Vision (3DV), 2019, IEEE, 240–248.
Zhu, Jieyun, Zhong, Zhimei, Li, Hongyuan, Ji, Pan, Pang, Jielong, Li, Bocheng, Zhang, Jianfeng, CT imaging features of 4121 patients with COVID-19: A meta-analysis. J. Med. Virol. 92:7 (2020), 891–902.
Zoph, Barret, Vasudevan, Vijay, Shlens, Jonathon, Le, Quoc V, 2018. Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 8697–8710.