android testing; Model reuse; model-based testing; regression testing; update testing; Android testing; Automated testing; Functional correctness; Functional testing; Model based testing; Regression testing; Testing technique; Update testing; Visual inspection; Software
Abstract :
[en] Although App updates are frequent and software engineers would like to verify updated features only, automated testing techniques verify entire Apps and are thus wasting resources. We present Continuous Adaptation of Learned Models (CALM), an automated App testing approach that efficiently test App updates by adapting App models learned when automatically testing previous App versions. CALM focuses on functional testing. Since functional correctness can be mainly verified through the visual inspection of App screens, CALM minimizes the number of App screens to be visualized by software testers while maximizing the percentage of updated methods and instructions exercised. Our empirical evaluation shows that CALM exercises a significantly higher proportion of updated methods and instructions than six state-of-the-art approaches, for the same maximum number of App screens to be visually inspected. Further, in common update scenarios, where only a small fraction of methods are updated, CALM is even quicker to outperform all competing approaches in a more significant way.
Research center :
Interdisciplinary Centre for Security, Reliability and Trust (SnT) > SVV - Software Verification and Validation
Disciplines :
Computer science
Author, co-author :
NGO, Chanh Duc ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust > SVV > Team Fabrizio PASTORE
PASTORE, Fabrizio ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
Briand, Lionel ; School of EECS, University of Ottawa, Ottawa, Canada ; Lero SFI Centre for Software Research, University of Limerick, Limerick, Ireland
External co-authors :
yes
Language :
English
Title :
Testing Updated Apps by Adapting Learned Models
Publication date :
29 June 2024
Journal title :
ACM Transactions on Software Engineering and Methodology
Huawei Technologies Co., Ltd, China NSERC Discovery and Canada Research Chair programs
Funding text :
This project has received funding from Huawei Technologies Co., Ltd, China, and by the NSERC Discovery and Canada Research Chair programs. Experiments presented in this article were carried out using the Grid\u20195000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr).This project has received funding from Huawei Technologies Co., Ltd, China, and by the NSERC Discovery and Canada Research Chair programs.
[n.d.]. Activity Diary - Github. Retrieved December 10, 2023 from https://github.com/ramack/ActivityDiary.
[n.d.]. Amaze File Manager - Github. Retrieved December 10, 2023 from https://github.com/TeamAmaze/ AmazeFileManager. Last visited: 12/10/2023.
Kevin Allix, Tegawendé F. Bissyandé, Jacques Klein, and Yves Le Traon. 2016. AndroZoo: Collecting millions of android apps for the research community. In Proceedings of the 13th International Conference on Mining Software Repositories (Austin, Texas). ACM, New York, NY, USA, 468–471. DOI:https://doi.org/10.1145/2901739.2903508
Android.com. 2022. Monkey - Android UI/application exerciser. Retrieved January 31, 2023 from https://developer.android.com/studio/test/other-testing-tools/monkey. Last visited: 01/31/2023.
Appium.io. 2023. Appium Documentation. Retrieved July 31, 2023 from http://appium.io. Last visited: 07/31/2023.
Nataniel P. Borges Jr., Jenny Hotzkow, and Andreas Zeller. 2018. DroidMate-2: A platform for android test generation. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (Montpellier, France). ACM, New York, NY, USA, 916–919. DOI:https://doi.org/10.1145/3238147.3240479
M.G.J. Brand, van den, Z. Protic, and T. Verhoeff. 2011. RCVDiff - a Stand-alone Tool for Representation, Calculation and Visualization of Model Differences. Technical Report. United States.
Paolo Calciati, Konstantin Kuznetsov, Xue Bai, and Alessandra Gorla. 2018. What did really change with the new release of the app?. In Proceedings of the 15th International Conference on Mining Software Repositories (Gothenburg, Sweden). Association for Computing Machinery, New York, NY, USA, 142–152. DOI:https://doi.org/10.1145/3196398. 3196449
Nana Chang, Linzhang Wang, Yu Pei, Subrota K. Mondal, and Xuandong Li. 2018. Change-based test script maintenance for android apps. In Proceedings of the 2018 IEEE International Conference on Software Quality, Reliability and Security. DOI:https://doi.org/10.1109/QRS.2018.00035
Wontae Choi, Koushik Sen, George Necula, and Wenyu Wang. 2018. DetReduce: Minimizing android GUI test suites for regression testing. In Proceedings of the 40th International Conference on Software Engineering (Gothenburg, Sweden). Association for Computing Machinery, New York, NY, USA, 445–455. DOI:https://doi.org/10.1145/3180155.3180173
Shauvik Roy Choudhary, Alessandra Gorla, and Alessandro Orso. 2015. Automated test input generation for android: Are we there yet? (E). In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering. IEEE Computer Society, Washington, DC, USA, 429–440. DOI:https://doi.org/10.1109/ASE.2015.89
Sergio Di Martino, Anna Rita Fasolino, Luigi Libero Lucio Starace, and Porfirio Tramontana. 2020. Comparing the effectiveness of capture and replay against automatic input generation for android graphical user interface testing. Software Testing Verification and Reliability 31, 3 (2020), 1–27. DOI:https://doi.org/10.1002/stvr.1754
Daniel Domínguez-Álvarez and Alessandra Gorla. 2019. Release practices for IOS and android apps. In Proceedings of the 3rd ACM SIGSOFT International Workshop on App Market Analytics (Tallinn, Estonia). Association for Computing Machinery, New York, NY, USA, 15–18. DOI:https://doi.org/10.1145/3340496.3342762
Zhen Dong, Marcel Böhme, Lucia Cojocaru, and Abhik Roychoudhury. 2020. Time-travel testing of android apps. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (Seoul, South Korea). Association for Computing Machinery, New York, NY, USA, 481–492. DOI:https://doi.org/10.1145/3377811.3380402
Zebao Gao, Zhenyu Chen, Yunxiao Zou, and Atif M. Memon. 2016. SITAR: GUI test script repair. IEEE Transactions on Software Engineering 42, 2 (2016), 170–186. DOI:https://doi.org/10.1109/TSE.2015.2454510
Zebao Gao, Chunrong Fang, and Atif M. Memon. 2015. Pushing the limits on automation in GUI regression testing. In Proceedings of the 2015 IEEE 26th International Symposium on Software Reliability Engineering. 565–575. DOI:https: //doi.org/10.1109/ISSRE.2015.7381848
Tianxiao Gu, Chengnian Sun, Xiaoxing Ma, Chun Cao, Chang Xu, Yuan Yao, Qirun Zhang, Jian Lu, and Zhendong Su. 2019. Practical GUI testing of android applications via model abstraction and refinement. In Proceedings of the 41st International Conference on Software Engineering (Montreal, Quebec, Canada). IEEE Press, 269–280. DOI:https: //doi.org/10.1109/ICSE.2019.00042
Wunan Guo, Zhen Dong, Liwei Shen, Wei Tian, Ting Su, and Xin Peng. 2022. Detecting and fixing data loss issues in android apps. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (Virtual, South Korea). Association for Computing Machinery, New York, NY, USA, 605–616. DOI:https://doi.org/10. 1145/3533767.3534402
Wunan Guo, Zhen Dong, Liwei Shen, Wei Tian, Ting Su, and Xin Peng. 2022. Detecting and fixing data loss issues in android apps. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (Virtual, South Korea). Association for Computing Machinery, New York, NY, USA, 605–616. DOI:https://doi.org/10. 1145/3533767.3534402
Wunan Guo, Liwei Shen, Ting Su, Xin Peng, and Weiyang Xie. 2020. Improving automated GUI exploration of android apps via static dependency analysis. Proceedings - 2020 IEEE International Conference on Software Maintenance and Evolution. 557–568. DOI: https://doi.org/10.1109/ICSME46990.2020.00059
Yavuz Koroglu, Alper Sen, Ozlem Muslu, Yunus Mete, Ceyda Ulker, Tolga Tanriverdi, and Yunus Donmez. 2018. QBE: QLearning-based exploration of android applications. In Proceedings of the 2018 IEEE 11th International Conference on Software Testing, Verification and Validation. https://doi.org/10.1109/ICST.2018.00020
Cong Li, Yanyan Jiang, and Chang Xu. 2022. Cross-device record and replay for android apps. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Singapore, Singapore). Association for Computing Machinery, New York, NY, USA, 395–407. DOI:https://doi.org/10. 1145/3540250.3549083
Xiao Li, Nana Chang, Yan Wang, Haohua Huang, Yu Pei, Linzhang Wang, and Xuandong Li. 2017. ATOM: Automatic maintenance of GUI test scripts for evolving mobile applications. In Proceedings of the 2017 IEEE International Conference on Software Testing, Verification and Validation. 161–171. DOI:https://doi.org/10.1109/ICST.2017.22
Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2019. Humanoid: A deep learning-based approach to automated black-box android app testing. In Proceedings of the 2019 34th IEEE/ACM International Conference on Automated Software Engineering. 1070–1073. DOI:https://doi.org/10.1109/ASE.2019.00104
Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2019. Humanoid: A deep learning-based approach to automated black-box android app testing. In Proceedings of the 2019 34th IEEE/ACM International Conference on Automated Software Engineering. DOI:https://doi.org/10.1109/ASE.2019.00104
M. Linares-Vasquez, K. Moran, and D. Poshyvanyk. 2017. Continuous, evolutionary and large-scale: A new perspective for automated mobile app testing. In Proceedings of the 2017 IEEE International Conference on Software Maintenance and Evolution. 399–410. DOI:https://doi.org/10.1109/ICSME.2017.27
Zhengwei Lv, Chao Peng, Zhao Zhang, Ting Su, Kai Liu, and Ping Yang. 2023. Fastbot2: Reusable automated model-based GUI testing for android enhanced by reinforcement learning. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (Rochester, MI, USA). Association for Computing Machinery, New York, NY, USA, 5 pages. DOI:https://doi.org/10.1145/3551349.3559505
Ke Mao, Mark Harman, and Yue Jia. 2016. Sapienz: Multi-objective automated testing for android applications. In Proceedings of the 25th International Symposium on Software Testing and Analysis (Saarbrücken, Germany). ACM, New York, NY, USA, 94–105. DOI:https://doi.org/10.1145/2931037.2931054
Stuart Mcilroy, Nasir Ali, and Ahmed E. Hassan. 2016. Fresh apps: An empirical study of frequently-updated mobile apps in the google play store. Empirical Software Engineering 21, 3 (2016), 1346–1370. DOI:https://doi.org/10.1007/ s10664-015-9388-2
Chanh Duc Ngo, Fabrizio Pastore, and Lionel Briand. 2022. Automated, cost-effective, and update-driven app testing. ACM Transactions on Software Engineering and Methodology 31, 4 (2022), 51 pages. DOI:https://doi.org/10.1145/ 3502297
Minxue Pan, An Huang, Guoxin Wang, Tian Zhang, and Xuandong Li. 2020. Reinforcement learning based curiosity-driven testing of android applications. Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. DOI:https://doi.org/10.1145/3395363.3397354
Minxue Pan, Tongtong Xu, Yu Pei, Zhong Li, Tian Zhang, and Xuandong Li. 2020. GUI-guided test script repair for mobile apps. IEEE Transactions on Software Engineering 5589, c (2020), 1–1. DOI:https://doi.org/10.1109/tse.2020. 3007664
F. Pastore, L. Mariani, and G. Fraser. 2013. CrowdOracles: Can the crowd solve the oracle problem? In Proceedings of the 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation. 342–351.
Chao Peng, Zhao Zhang, Zhengwei Lv, and Ping Yang. 2022. MUBot: Learning to test large-scale commercial android apps like a human. In Proceedings of the 2022 IEEE International Conference on Software Maintenance and Evolution. 543–552. DOI:https://doi.org/10.1109/ICSME55016.2022.00074
Oliviero Riganelli, Simone Paolo Mottadelli, Claudio Rota, Daniela Micucci, and Leonardo Mariani. 2020. Data loss detector: Automatically revealing data loss bugs in android apps. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, USA, July 18–22, 2020. Sarfraz Khurshid and Corina S. Pasareanu (Eds.), ACM, 141–152. DOI:https://doi.org/10.1145/3395363.3397379
Andrea Romdhana, Alessio Merlo, Mariano Ceccato, and Paolo Tonella. 2022. Deep reinforcement learning for black-box testing of android apps. ACM Transactions on Software Engineering and Methodology 31 4 (2022), 1–29. DOI:https: //doi.org/10.1145/3502868
Aman Sharma and Rupesh Nasre. 2019. QADroid: Regression event selection for android applications. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (Beijing, China). Association for Computing Machinery, New York, NY, USA, 66–77. DOI:https://doi.org/10.1145/3293882.3330550
Ting Su, Guozhu Meng, Yuting Chen, Ke Wu, Weiming Yang, Yao Yao, Geguang Pu, Yang Liu, and Zhendong Su. 2017. Guided, stochastic model-based GUI testing of android apps. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (Paderborn, Germany). ACM, New York, NY, USA, 245–256. DOI:https://doi.org/ 10.1145/3106237.3106298
Ting Su, Yichen Yan, Jue Wang, Jingling Sun, Yiheng Xiong, Geguang Pu, Ke Wang, and Zhendong Su. 2021. Fully automated functional fuzzing of android apps for detecting non-crashing logic bugs. Proceedings of the ACM on Programming Languages 5, OOPSLA (2021), 31 pages. DOI:https://doi.org/10.1145/3485533
Jingling Sun, Ting Su, Junxin Li, Zhen Dong, Geguang Pu, Tao Xie, and Zhendong Su. 2021. Understanding and finding system setting-related defects in android apps. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis (Virtual, Denmark). Association for Computing Machinery, New York, NY, USA, 204–215. DOI:https://doi.org/10.1145/3460319.3464806
Saghar Talebipour, Yixue Zhao, Luka Dojcilović, Chenggang Li, and Nenad Medvidović. 2021. UI test migration across mobile platforms. In Proceedings of the 2021 36th IEEE/ACM International Conference on Automated Software Engineering. 756–767. DOI:https://doi.org/10.1109/ASE51524.2021.9678643
Porfirio Tramontana, Domenico Amalfitano, Nicola Amatucci, and Anna Rita Fasolino. 2019. Automated functional testing of mobile applications: A systematic mapping study. Software Quality Journal 27, 1 (2019), 149–201. DOI:https: //doi.org/10.1007/s11219-018-9418-6
Mark van den Brand, Zvezdan Protić, and Tom Verhoeff. 2010. Fine-grained metamodel-assisted model comparison. In Proceedings of the 1st International Workshop on Model Comparison in Practice. ACM, New York, New York, USA. DOI:https://doi.org/10.1145/1826147.1826152
András Vargha and Harold D. Delaney. 2000. A critique and improvement of the CL common language effect size statistics of McGraw and wong. Journal of Educational and Behavioral Statistics 25, 2 (2000), 101–132. DOI:https://doi. org/10.3102/10769986025002101 arXiv:https://doi.org/10.3102/10769986025002101
Jue Wang, Yanyan Jiang, Ting Su, Shaohua Li, Chang Xu, Jian Lu, and Zhendong Su. 2022. Detecting non-crashing functional bugs in android apps via deep-state differential analysis. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Singapore, Singapore). Association for Computing Machinery, New York, NY, USA, 434–446. DOI:https://doi.org/10.1145/3540250.3549170
Jue Wang, Yanyan Jiang, Chang Xu, Chun Cao, Xiaoxing Ma, and Jian Lu. 2020. ComboDroid: Generating high-quality test inputs for android apps via use case combinations. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (Seoul, South Korea). Association for Computing Machinery, New York, NY, USA, 469–480. DOI:https://doi.org/10.1145/3377811.3380382
Wenyu Wang, Dengfeng Li, Wei Yang, Yurui Cao, Zhenwen Zhang, Yuetang Deng, and Tao Xie. 2018. An empirical study of android test generation tools in industrial cases. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (Montpellier, France). ACM, New York, NY, USA, 738–748. DOI:https://doi.org/10. 1145/3238147.3240465
Yiheng Xiong, Mengqian Xu, Ting Su, Jingling Sun, Jue Wang, He Wen, Geguang Pu, Jifeng He, and Zhendong Su. 2023. An empirical study of functional bugs in android apps. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (Seattle, WA, USA). Association for Computing Machinery, New York, NY, USA, 1319–1331. DOI:https://doi.org/10.1145/3597926.3598138
Tongtong Xu, Minxue Pan, Yu Pei, Guiyin Li, Xia Zeng, Tian Zhang, Yuetang Deng, and Xuandong Li. 2021. GUIDER: GUI structure and vision co-guided test script repair for android apps. ISSTA 2021 - Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis. 191–203. DOI:https://doi.org/10.1145/3460319. 3464830
Shengqian Yang, Haowei Wu, Hailong Zhang, Yan Wang, Chandrasekar Swaminathan, Dacong Yan, and Atanas Rountev. 2018. Static window transition graphs for android. International Journal of Automated Software Engineering 25, 4 (2018), 833–873.
Faraz Yazdani Banafshe Daragh and Sam Malek. 2021. Deep GUI: Black-box GUI input generation with deep learning. In Proceedings of the 2021 36th IEEE/ACM International Conference on Automated Software Engineering. 905–916. DOI:https://doi.org/10.1109/ASE51524.2021.9678778
Minghe Yu, Guoliang Li, Dong Deng, and Jianhua Feng. 2016. String similarity search and join: A survey. Frontiers of Computer Science 10, 3 (2016), 399–417. DOI:https://doi.org/10.1007/s11704-015-5900-5
Yifei Zhang, Yulei Sui, and Jingling Xue. 2018. Launch-mode-aware context-sensitive activity transition analysis. In Proceedings of the 40th International Conference on Software Engineering (Gothenburg, Sweden). Association for Computing Machinery, New York, NY, USA, 598–608. DOI:https://doi.org/10.1145/3180155.3180188
