The effect of dataset confounding on predictions of deep neural networks for medical imaging

in Vol. 3 (2022): Proceedings of the Northern Lights Deep Learning Workshop 2022 (2022, April 18)

The use of Convolutional Neural Networks (CNN) in medical imaging has often outperformed previous solutions and even specialists, becoming a promising technology for Computer-aided-Diagnosis (CAD) systems ... [more ▼]

The use of Convolutional Neural Networks (CNN) in medical imaging has often outperformed previous solutions and even specialists, becoming a promising technology for Computer-aided-Diagnosis (CAD) systems. However, recent works suggested that CNN may have poor generalisation on new data, for instance, generated in different hospitals. Uncontrolled confounders have been proposed as a common reason. In this paper, we experimentally demonstrate the impact of confounding data in unknown scenarios. We assessed the effect of four confounding configurations: total, strong, light and balanced. We found the confounding effect is especially prominent in total confounder scenarios, while the effect on light and strong confounding scenarios may depend on the dataset robustness. Our findings indicate that the confounding effect is independent of the architecture employed. These findings might explain why models can report good metrics during the development stage but fail to translate to real-world settings. We highlight the need for thorough consideration of these commonly unattended aspects, to develop safer CNN-based CAD systems. [less ▲]

Towards Generalizable Machine Learning for Chest X-ray Diagnosis with Multi-task learning

E-print/Working paper (2022)

Clinicians use chest radiography (CXR) to diagnose common pathologies. Automated classification of these diseases can expedite analysis workflow, scale to growing numbers of patients and reduce healthcare ... [more ▼]

Clinicians use chest radiography (CXR) to diagnose common pathologies. Automated classification of these diseases can expedite analysis workflow, scale to growing numbers of patients and reduce healthcare costs. While research has produced classification models that perform well on a given dataset, the same models lack generalization on different datasets. This reduces confidence that these models can be reliably deployed across various clinical settings. We propose an approach based on multitask learning to improve model generalization. We demonstrate that learning a (main) pathology together with an auxiliary pathology can significantly impact generalization performance (between -10% and +15% AUC-ROC). A careful choice of auxiliary pathology even yields competitive performance with state-of-the-art models that rely on fine-tuning or ensemble learning, using between 6% and 34% of the training data that these models required. We, further, provide a method to determine what is the best auxiliary task to choose without access to the target dataset. Ultimately, our work makes a big step towards the creation of CXR diagnosis models applicable in the real world, through the evidence that multitask learning can drastically improve generalization. [less ▲]

The need of standardised metadata to encode causal relationships: Towards safer data-driven machine learning biological solutions

Scientific Conference (2021, November 16)

In this paper, we discuss the importance of considering causal relations in the development of machine learning solutions to prevent factors hampering the robustness and generalisation capacity of the ... [more ▼]

In this paper, we discuss the importance of considering causal relations in the development of machine learning solutions to prevent factors hampering the robustness and generalisation capacity of the models, such as induced biases. This issue often arises when the algorithm decision is affected by confounding factors. In this work, we argue that the integration of causal relationships can identify potential confounders. We call for standardised meta-information practices as a crucial step for proper machine learning solutions development, validation, and data sharing. Such practices include detailing the dataset generation process, aiming for automatic integration of causal relationships. [less ▲]

Automatic Detection of Nigrosome Degeneration in Susceptibility-Weighted MRI for Computer-Aided Diagnosis of Parkinson’s Disease Using Machine Learning

in Movement Disorders (2020, September 12)

Objective: Automatize the detection of ‘swallow-tail’ appearance in substantia nigra dopaminergic neurons using MRI for more robust tests on Parkinson’s disease (PD) diagnosis. Background: Differential ... [more ▼]

Objective: Automatize the detection of ‘swallow-tail’ appearance in substantia nigra dopaminergic neurons using MRI for more robust tests on Parkinson’s disease (PD) diagnosis. Background: Differential diagnosis of PD is challenging even in specialized centers. The use of imaging techniques can be bene cial for the diagnosis. Although DaTSCAN has been proven to be clinically useful, it is not widely available and has radiation risk and high-cost associated. Therefore, MRI scans for PD diagnosis offer several advantages over DaTSCAN [1]. Recent literature shows strong evidence of high diagnostic accuracy using the ‘swallow-tail’ shape of the dorsolateral substantia nigra in 3T – SWI [2]. Nevertheless, the majority of such studies rely on the subjective opinion of experts and manual methods for the analysis to assess the accuracy of these features. Alternatively, we propose a fully automated solution to evaluate the absence or presence of this feature for computer-aided diagnosis (CAD) of PD. Method: Restrospective study of 27 PD and 18 non-PD was conducted, including standard high-resolution 3D MRI – T1 & SWI sequences (additionally, T2 scans were used to increase the registration references). Firstly, spatial registration and normalization of the images were performed. Then, the ROI was extracted using atlas references. Finally, a supervised machine learning model was built using 5-fold-within-5-fold nested cross-validation. Results: Preliminary results show signi cant sensitivity (0.92) and ROC AUC (0.82), allowing for automated classi cation of patients based on swallow-tail biomarker from MRI. Conclusion: Detection of nigrosome degeneration (swallow-tail biomarker) in accessible brain imaging techniques can be automatized with signi cant accuracy, allowing for computer-aided PD diagnosis. References: [1] Schwarz, S. T., Xing, Y., Naidu, S., Birchall, J., Skelly, R., Perkins, A., ... & Gowland, P. (2017). Protocol of a single group prospective observational study on the diagnostic value of 3T susceptibility weighted MRI of nigrosome-1 in patients with parkinsonian symptoms: the N3iPD study (nigrosomal iron imaging in Parkinson’s disease). BMJ open, 7(12), e016904. [2] – Schwarz, S. T., Afzal, M., Morgan, P. S., Bajaj, N., Gowland, P. A., & Auer, D. P. (2014). The ‘swallow tail’ appearance of the healthy nigrosome –a new accurate test of Parkinson’s disease: a case-control and retrospective cross-sectional MRI study at 3T. PloS one, 9(4). [less ▲]

From tech to bench: Deep Learning pipeline for image segmentation of high-throughput high-content microscopy data

Poster (2019, November 29)

Automation of biological image analysis is essential to boost biomedical research. The study of complex diseases such as neurodegenerative diseases calls for big amounts of data to build models towards ... [more ▼]

Automation of biological image analysis is essential to boost biomedical research. The study of complex diseases such as neurodegenerative diseases calls for big amounts of data to build models towards precision medicine. Such data acquisition is feasible in the context of high-throughput screening in which the quality of the results relays on the accuracy of image analysis. Although the state-of-the-art solutions for image segmentation employ deep learning approaches, the high cost of manual data curation is hampering the real use in current biomedical research laboratories. Here, we propose a pipeline that employs deep learning not only to conduct accurate segmentation but also to assist with the creation of high-quality datasets in a less time-consuming solution for the experts. Weakly-labelled datasets are becoming a common alternative as a starting point to develop real-world solutions. Traditional approaches based on classical multimedia signal processing were employed to generate a pipeline specifically optimized for the high-throughput screening images of iPSC fused with rosella biosensor. Such pipeline produced good segmentation results but with several inaccuracies. We employed the weakly-labelled masks produced in this pipeline to train a multiclass semantic segmentation CNN solution based on U-net architecture. Since a strong class imbalance was detected between the classes, we employed a class sensitive cost function: Dice coe!cient. Next, we evaluated the accuracy between the weakly-labelled data and the trained network segmentation using double-blind tests conducted by experts in cell biology with experience in this type of images; as well as traditional metrics to evaluate the quality of the segmentation using manually curated segmentations by cell biology experts. In all the evaluations the prediction of the neural network overcomes the weakly-labelled data quality segmentation. Another big handicap that complicates the use of deep learning solutions in wet lab environments is the lack of user-friendly tools for non-computational experts such as biologists. To complete our solution, we integrated the trained network on a GUI built on MATLAB environment with non-programming requirements for the user. This integration allows conducting semantic segmentation of microscopy images in a few seconds. In addition, thanks to the patch-based approach it can be employed in images with different sizes. Finally, the human-experts can correct the potential inaccuracies of the prediction in a simple interactive way which can be easily stored and employed to re-train the network to improve its accuracy. In conclusion, our solution focuses on two important bottlenecks to translate leading-edge technologies in computer vision to biomedical research: On one hand, the effortless obtention of high-quality datasets with expertise supervision taking advantage of the proven ability of our CNN solution to generalize from weakly-labelled inaccuracies. On the other hand, the ease of use provided by the GUI integration of our solution to both segment images and interact with the predicted output. Overall this approach looks promising for fast adaptability to new scenarios. [less ▲]

Automated analysis of high-throughput screening high-content microscopy data with deep learning

Bachelor/master dissertation (2019)