Fine-grained Code Coverage Measurement in Automated Black-box Android Testing

in ACM Transactions on Software Engineering and Methodology (2020), 29(4), 1-35

Today, there are millions of third-party Android applications. Some of them are buggy or even malicious. To identify such applications, novel frameworks for automated black-box testing and dynamic ... [more ▼]

Today, there are millions of third-party Android applications. Some of them are buggy or even malicious. To identify such applications, novel frameworks for automated black-box testing and dynamic analysis are being developed by the Android community. Code coverage is one of the most common metrics for evaluating effectiveness of these frameworks. Furthermore, code coverage is used as a fitness function for guiding evolutionary and fuzzy testing techniques. However, there are no reliable tools for measuring fine-grained code coverage in black-box Android app testing. We present the Android Code coVerage Tool, ACVTool for short, that instruments Android apps and measures code coverage in the black-box setting at class, method and instruction granularity. ACVTool has successfully instrumented 96.9% of apps in our experiments. It introduces a negligible instrumentation time overhead, and its runtime overhead is acceptable for automated testing tools. We demonstrate practical value of ACVTool in a large-scale experiment with Sapienz, a state-of-art automated testing tool. Using ACVTool on the same cohort of apps, we have compared different coverage granularities applied by Sapienz in terms of the found amount of crashes. Our results show that none of the applied coverage granularities clearly outperforms others in this aspect. [less ▲]

Attack-Tree Series: A Case for Dynamic Attack Tree Analysis

in Proc.\ 6th International Workshop on Graphical Models for Security (GraMSec'19) (2020)

DEMO: An Effective Android Code Coverage Tool

Poster (2018, October 15)

The deluge of Android apps from third-party developers calls for sophisticated security testing and analysis techniques to inspect suspicious apps without accessing their source code. Code coverage is an ... [more ▼]

The deluge of Android apps from third-party developers calls for sophisticated security testing and analysis techniques to inspect suspicious apps without accessing their source code. Code coverage is an important metric used in these techniques to evaluate their effectiveness, and even as a fitness function to help achieving better results in evolutionary and fuzzy approaches. Yet, so far there are no reliable tools for measuring fine-grained bytecode coverage of Android apps. In this work we present ACVTool that instruments Android apps and measures the smali code coverage at the level of classes, methods, and instructions. Tool repository: https://github.com/pilgun/acvtool [less ▲]

New Directions in Attack Tree Research: Catching up with Industrial Needs

in Mauw, Sjouke (Ed.) Proceedings of the 4th International Workshop on Graphical Models for Security (2018, January)

Attack trees provide a systematic way of characterizing diverse system threats. Their strengths arise from the combination of an intuitive representation of possible attacks and availability of formal ... [more ▼]

Attack trees provide a systematic way of characterizing diverse system threats. Their strengths arise from the combination of an intuitive representation of possible attacks and availability of formal mathematical frameworks for analyzing them in a qualitative or a quantitative manner. Indeed, the mathematical frameworks have become a large focus of attack tree research. However, practical applications of attack trees in industry largely remain a tedious and error-prone exercise. Recent research directions in attack trees, such as attack tree generation, attempt to close this gap and to improve the attack tree state-of-thepractice. In this position paper we outline the recurrent challenges in manual tree design within industry, and we overview the recent research results in attack trees that help the practitioners. For the challenges that have not yet been addressed by the community, we propose new promising research directions. [less ▲]

Small changes, big changes: an updated view on the Android permission system

in Research in Attacks, Intrusions, and Defenses - 19th International Symposium, RAID 2016, Paris, France, September 19-21, 2016, Proceedings (2016, September)

Since the appearance of Android, its permission system was central to many studies of Android security. For a long time, the description of the architecture provided by Enck et al. was immutably used in ... [more ▼]

Since the appearance of Android, its permission system was central to many studies of Android security. For a long time, the description of the architecture provided by Enck et al. was immutably used in various research papers. The introduction of highly anticipated runtime permissions in Android 6.0 forced us to reconsider this model. To our surprise, the permission system evolved with almost every release. After analysis of 16 Android versions, we can con firm that the modi fications, especially introduced in Android 6.0, considerably impact the aptness of old conclusions and tools for newer releases. For instance, since Android 6.0 some signature permissions, previously granted only to apps signed with a platform certi cate, can be granted to third-party apps even if they are signed with a non-platform certi cate; many permissions considered before as threatening are now granted by default. In this paper, we review in detail the updated system, introduced changes, and their security implications. We highlight some bizarre behaviors, which may be of interest for developers and security researchers. We also found a number of bugs during our analysis, and provided patches to AOSP where possible. [less ▲]

Modelling Attack-defense Trees Using Timed Automata

in Proc. of FORMATS (2016)

Performing a thorough security risk assessment of an organisation has always been challenging, but with the increased reliance on outsourced and off-site third-party services, i.e., ``cloud services ... [more ▼]

Performing a thorough security risk assessment of an organisation has always been challenging, but with the increased reliance on outsourced and off-site third-party services, i.e., ``cloud services'', combined with internal (legacy) IT-infrastructure and -services, it has become a very difficult and time-consuming task. One of the traditional tools available to ease the burden of performing a security risk assessment and structure security analyses in general is attack trees, a tree-based formalism inspired by fault trees, a well-known formalism used in safety engineering. In this paper we study an extension of traditional attack trees, called attack-defense trees, in which not only the attacker's actions are modelled, but also the defensive actions taken by the attacked party. In this work we use the attack-defense tree as a goal an attacker wants to achieve, and separate the behaviour of the attacker and defender from the attack-defense-tree. We give a fully stochastic timed semantics for the behaviour of the attacker by introducing attacker profiles that choose actions probabilistically and execute these according to a probability density. Lastly, the stochastic semantics provides success probabilitites for individual actions. Furthermore, we show how to introduce costs of attacker actions. Finally, we show how to automatically encode it all with a network of timed automata, an encoding that enables us to apply state-of-the-art model checking tools and techniques to perform fully automated quantitative and qualitative analyses of the modelled system. [less ▲]

Attack trees for practical security assessment: ranking of attack scenarios with ADTool 2.0

in Quantitative Evaluation of Systems - 13th International Conference (2016)

Evaluation of Resource-based App Repackaging Detection in Android

in Proc. of NordSec (2016)

Android app repackaging threatens the health of application markets, as repackaged apps, besides stealing revenue for honest developers, are also a source of malware distribution. Techniques that rely on ... [more ▼]

Android app repackaging threatens the health of application markets, as repackaged apps, besides stealing revenue for honest developers, are also a source of malware distribution. Techniques that rely on visual similarity of Android apps recently emerged as a way to tackle the repackaging detection problem, as code-based detection techniques often fail in terms of effi ciency, and e ffectiveness when obfuscation is applied [19,21]. Among such techniques, the resource-based repackaging detection approach that compares sets of files included in apks has arguably the best performance [20,17,10]. Yet, this approach has not been previously validated on a dataset of repackaged apps. In this paper we report on our evaluation of the approach, and present substantial improvements to it. Our experiments show that the state-of-art tools applying this technique rely on too restrictive thresholds. Indeed, we demonstrate that a very low proportion of identical resource files in two apps is a reliable evidence for repackaging. Furthermore, we have shown that the Overlap similarity score performs better than the Jaccard similarity coe fficient used in previous works. By applying machine learning techniques, we give evidence that considering separately the included resource fi le types signi cantly improves the detection accuracy of the method. Experimenting with a balanced dataset of more than 2700 app pairs, we show that with our enhancements it is possible to achieve the F-measure of 0.9919. [less ▲]

Towards Black Box Testing of Android Apps

in Proc. of Software Assurance Workshop at the 10th International Conference on Availability, Reliability and Security (ARES) (2015, August)

Many state-of-art mobile application testing frameworks (e.g., Dynodroid, EvoDroid) enjoy Emma or other code coverage libraries to measure the coverage achieved. The underlying assumption for these ... [more ▼]

Many state-of-art mobile application testing frameworks (e.g., Dynodroid, EvoDroid) enjoy Emma or other code coverage libraries to measure the coverage achieved. The underlying assumption for these frameworks is availability of the app source code. Yet, application markets and security researchers face the need to test third-party mobile applications in the absence of the source code. There exists a number of frameworks both for manual and automated test generation that address this challenge. However, these frameworks often do not provide any statistics on the code coverage achieved, or provide coarse-grained ones like a number of activities or methods covered. At the same time, given two test reports generated by different frameworks, there is no way to understand which one achieved better coverage if the reported metrics were different (or no coverage results were provided). To address these issues we designed a framework called BBOXTESTER that is able to generate code coverage reports and produce uniform coverage metrics in testing without the source code. Security researchers can automatically execute applications exploiting current stateof- art tools, and use the results of our framework to assess if the security-critical code was covered by the tests. In this paper we report on design and implementation of BBOXTESTER and assess its efficiency and effectiveness. [less ▲]

FSquaDRA: Fast Detection of Repackaged Applications

in Data and Applications Security and Privacy XXVIII (2014, July)

The ease of Android applications repackaging and proliferation of application clones in Google Play and other markets call for new effective techniques to detect repackaged code and combat distribution of ... [more ▼]

The ease of Android applications repackaging and proliferation of application clones in Google Play and other markets call for new effective techniques to detect repackaged code and combat distribution of cloned applications. Today all existing techniques for repackaging detection are based on code similarity or feature (e.g., permission set) similarity evaluation. We propose a new approach to detect repackaging based on the resource files available in application packages. Our tool called FSquaDRA performs a quick pairwise application comparison (full pairwise comparison for 55,000 applications in just 80 hours on a laptop), as it measures how many identical resources are present inside both packages under analysis. The intuition behind our approach is that malicious repackaged applications still need to maintain the “look and feel” of the originals by including the same images and other resource files, even though they might have additional code included or some of the original code removed. To evaluate the reliability of our approach we perform a comparison of the FSquaDRA similarity scores with the code-based similarity scores of AndroGuard for a dataset of randomly selected application pairs, and our results demonstrate strong positive correlation of the FSquaDRA resource-based score with the code-based similarity score. [less ▲]