Paper published in a book (Scientific congresses, symposiums and conference proceedings)
Towards Exploring the Limitations of Active Learning: An Empirical Study
Hu, Qiang; Guo, Yuejun; Cordy, Maxime et al.
2021In The 36th IEEE/ACM International Conference on Automated Software Engineering.
Peer reviewed
 

Files


Full Text
ASE2021-ALempirical.pdf
Author postprint (2.06 MB)
Download

All documents in ORBilu are protected by a user license.

Send to



Details



Keywords :
deep learning; data selection; active learning; empirical study
Abstract :
[en] Deep neural networks (DNNs) are being increasingly deployed as integral parts of software systems. However, due to the complex interconnections among hidden layers and massive hyperparameters, DNNs require being trained using a large number of labeled inputs, which calls for extensive human effort for collecting and labeling data. Spontaneously, to alleviate this growing demand, a surge of state-of-the-art studies comes up with different metrics to select a small yet informative dataset for the model training. These research works have demonstrated that DNN models can achieve competitive performance using a carefully selected small set of data. However, the literature lacks proper investigation of the limitations of data selection metrics, which is crucial to apply them in practice. In this paper, we fill this gap and conduct an extensive empirical study to explore the limits of selection metrics. Our study involves 15 selection metrics evaluated over 5 datasets (2 image classification tasks and 3 text classification tasks), 10 DNN architectures, and 20 labeling budgets (ratio of training data being labeled). Our findings reveal that, while selection metrics are usually effective in producing accurate models, they may induce a loss of model robustness (against adversarial examples) and resilience to compression. Overall, we demonstrate the existence of a trade-off between labeling effort and different model qualities. This paves the way for future research in devising selection metrics considering multiple quality criteria.
Disciplines :
Computer science
Author, co-author :
Hu, Qiang ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SerVal
Guo, Yuejun ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SerVal
Cordy, Maxime  ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SerVal
Xiaofei, Xie
Ma, Wei ;  University of Luxembourg > Faculty of Science, Technology and Medecine (FSTM)
Papadakis, Mike ;  University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)
Le Traon, Yves ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SerVal
External co-authors :
yes
Language :
English
Title :
Towards Exploring the Limitations of Active Learning: An Empirical Study
Publication date :
2021
Event name :
The 36th IEEE/ACM International Conference on Automated Software Engineering (ASE 2021)
Event date :
from 15-11-2021 to 19-11-2021
Main work title :
The 36th IEEE/ACM International Conference on Automated Software Engineering.
Peer reviewed :
Peer reviewed
Focus Area :
Security, Reliability and Trust
FnR Project :
FNR12669767 - Testing Self-learning Systems, 2018 (01/09/2019-31/08/2022) - Yves Le Traon
Name of the research project :
CORE project C18/IS/12669767/ STELLAR/LeTraon
Funders :
FNR - Fonds National de la Recherche [LU]
Available on ORBilu :
since 15 October 2021

Statistics


Number of views
420 (101 by Unilu)
Number of downloads
828 (109 by Unilu)

Scopus citations®
 
7
Scopus citations®
without self-citations
2

Bibliography


Similar publications



Contact ORBilu