Large-scale Machine Learning-based Malware Detection: Confronting the "10-fold Cross Validation" Scheme with Reality

To address the issue of malware detection, researchers have
recently started to investigate the capabilities of machine-
learning techniques for proposing effective approaches. Sev-
eral promising results were recorded in the literature, many
approaches being assessed with the common “10-Fold cross
validation” scheme. This paper revisits the purpose of mal-
ware detection to discuss the adequacy of the “10-Fold” scheme
for validating techniques that may not perform well in real-
ity. To this end, we have devised several Machine Learning
classifiers that rely on a novel set of features built from ap-
plications’ CFGs. We use a sizeable dataset of over 50,000
Android applications collected from sources where state-of-
the art approaches have selected their data. We show that
our approach outperforms existing machine learning-based
approaches. However, this high performance on usual-size
datasets does not translate in high performance in the wild.