Article (Scientific journals)
An Empirical Study on Data Distribution-Aware Test Selection for Deep Learning Enhancement
Hu, Qiang; Guo, Yuejun; Cordy, Maxime et al.
2022In ACM Transactions on Software Engineering and Methodology
Peer reviewed
 

Files


Full Text
TOSEM_DAT.pdf
Author preprint (1.81 MB)
Download

All documents in ORBilu are protected by a user license.

Send to



Details



Keywords :
deep learning testing; test selection; data distribution
Abstract :
[en] Similar to traditional software that is constantly under evolution, deep neural networks (DNNs) need to evolve upon the rapid growth of test data for continuous enhancement, e.g., adapting to distribution shift in a new environment for deployment. However, it is labor-intensive to manually label all the collected test data. Test selection solves this problem by strategically choosing a small set to label. Via retraining with the selected set, DNNs will achieve competitive accuracy. Unfortunately, existing selection metrics involve three main limitations: 1) using different retraining processes; 2) ignoring data distribution shifts; 3) being insufficiently evaluated. To fill this gap, we first conduct a systemically empirical study to reveal the impact of the retraining process and data distribution on model enhancement. Then based on our findings, we propose a novel distribution-aware test (DAT) selection metric. Experimental results reveal that retraining using both the training and selected data outperforms using only the selected data. None of the selection metrics perform the best under various data distributions. By contrast, DAT effectively alleviates the impact of distribution shifts and outperforms the compared metrics by up to 5 times and 30.09% accuracy improvement for model enhancement on simulated and in-the-wild distribution shift scenarios, respectively.
Disciplines :
Computer science
Author, co-author :
Hu, Qiang ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SerVal
Guo, Yuejun ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SerVal
Cordy, Maxime  ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SerVal
Xie, Xiaofei;  Singapore Management University
Ma, Lei;  University of Alberta
Papadakis, Mike ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > Computer Science and Communications Research Unit (CSC)
Le Traon, Yves ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SerVal
External co-authors :
yes
Language :
English
Title :
An Empirical Study on Data Distribution-Aware Test Selection for Deep Learning Enhancement
Publication date :
2022
Journal title :
ACM Transactions on Software Engineering and Methodology
Peer reviewed :
Peer reviewed
FnR Project :
FNR12669767 - Testing Self-learning Systems, 2018 (01/09/2019-31/08/2022) - Yves Le Traon
Name of the research project :
CORE project C18/IS/12669767/STELLAR/LeTraon
Funders :
FNR - Fonds National de la Recherche [LU]
Available on ORBilu :
since 12 February 2022

Statistics


Number of views
515 (125 by Unilu)
Number of downloads
304 (36 by Unilu)

Scopus citations®
 
15
Scopus citations®
without self-citations
10
OpenCitations
 
0
WoS citations
 
13

Bibliography


Similar publications



Contact ORBilu