Form filling; Data entry forms; Machine Learning; Software data quality; User interfaces
Résumé :
[en] Users frequently interact with software systems through data entry forms. However, form filling is time-consuming and error-prone. Although several techniques have been proposed to auto-complete or pre-fill
fields in the forms, they provide limited support to help users fill categorical fields, i.e., fields that require
users to choose the right value among a large set of options.
In this paper, we propose LAFF, a learning-based automated approach for filling categorical fields in data
entry forms. LAFF first builds Bayesian Network models by learning field dependencies from a set of historical
input instances, representing the values of the fields that have been filled in the past. To improve its learning
ability, LAFF uses local modeling to effectively mine the local dependencies of fields in a cluster of input
instances. During the form filling phase, LAFF uses such models to predict possible values of a target field,
based on the values in the already-filled fields of the form and their dependencies; the predicted values
(endorsed based on field dependencies and prediction confidence) are then provided to the end-user as a list of
suggestions.
We evaluated LAFF by assessing its effectiveness and efficiency in form filling on two datasets, one of
them proprietary from the banking domain. Experimental results show that LAFF is able to provide accurate
suggestions with a Mean Reciprocal Rank value above 0.73. Furthermore, LAFF is efficient, requiring at most
317 ms per suggestion.
Centre de recherche :
Interdisciplinary Centre for Security, Reliability and Trust (SnT) > SVV - Software Verification and Validation
Disciplines :
Sciences informatiques
Auteur, co-auteur :
BELGACEM, Hichem ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
Li, Xiaochen; Dalian University of Technology ; University of Luxembourg > Interdisciplinary Center for Security, Reliability and Trust
BIANCULLI, Domenico ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
BRIAND, Lionel ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
Co-auteurs externes :
yes
Langue du document :
Anglais
Titre :
A Machine Learning Approach for Automated Filling of Categorical Fields in Data Entry Forms
Date de publication/diffusion :
mars 2023
Titre du périodique :
ACM Transactions on Software Engineering and Methodology
ISSN :
1049-331X
Maison d'édition :
Association for Computing Machinery (ACM), Etats-Unis
R. B. Aggarwal, Amit Dhawan, and Jay Shankar Kumar. 2007. Database-centric development of menus and graphic user interfaces. Defence Science Journal 57, 1 (2007), 133.
Pierre A. Akiki, Arosha K. Bandara, and Yijun Yu. 2016. Engineering adaptive model-driven user interfaces. IEEE Transactions on Software Engineering 42, 12 (2016), 1118-1147.
Stamatios-Aggelos N. Alexandropoulos, Sotiris B. Kotsiantis, and Michael N. Vrahatis. 2019. Data preprocessing in predictive data mining. Knowledge Engineering Review 34 (2019), e1.
Alnur Ali and Chris Meek. 2009. Predictive Models of Form Filling. Technical Report MSR-TR-2009-1. Microsoft Research. https://www.microsoft.com/en-us/research/publication/predictive-models-of-form-filling/.
American Medical News. 2005. Data entry is a top cause of medication errors. https://amednews.com/article /20050124/profession/301249959/4/.
Yuan An, Xiaohua Hu, and Il-Yeol Song. 2012. Learning to discover complex mappings from web forms to ontologies. In Proceedings of the 21st ACM International Conference on Information and Knowledge Management. ACM, New York, NY, 1253-1262.
Android API Reference. 2021. Android View Autofill. https://developer.android.com/reference/kotlin/android/view/ autofill/package-summary.
Ankur Ankan and Abinash Panda. 2015. Pgmpy: Probabilistic graphicalmodels using python. In Proc. Python in Science Conference (SCIPY'15). SCIPY, Austin, TX, 6-11.
Samur Araujo, Qi Gao, Erwin Leonardi, and Geert-Jan Houben. 2010. Carbon: Domain-independent automatic web form filling. In Proc. International Conference on Web Engineering (ICWE'10) (LNCS, Vol. 6189). Springer, Berlin, Germany, 292-306.
Iman Avazpour, Teerat Pitakrat, Lars Grunske, and John Grundy. 2014. Dimensions and metrics for evaluating recommendation systems. In Recommendation Systems in Software Engineering. Springer, Berlin, Germany, 245-273.
Bank for International Settlements. 2003. General Guide to Account Opening and Customer Identification. https: //www.bis.org/publ/bcbs85annex.htm.
Tanya Barrett, Karen Clark, Robert Gevorgyan, Vyacheslav Gorelenkov, Eugene Gribov, Ilene Karsch-Mizrachi, Michael Kimelman, Kim D. Pruitt, Sergei Resenchuk, Tatiana Tatusova, et al. 2012. BioProject and BioSample databases at NCBI: Facilitating capture and organization of metadata. Nucleic Acids Research 40, D1 (2012), D57-D63.
Leo Breiman, Jerome Friedman, Charles J. Stone, and Richard A. Olshen. 1984. Classification and Regression Trees. CRC Press, Boca Raton, FL.
Loli Burgueño, Robert Clarisó, Shuai Li, Sébastien Gérard, and Jordi Cabot. 2021. A NLP-based architecture for the autocompletion of partial domain models. International Conference on Advanced Information Systems Engineering (CAiSE'21) (LNCS). Springer, Berlin, Germany, 15 pages.
Valeria Cardellini, Michele Colajanni, and Philip S. Yu. 2000. Geographic load balancing for scalable distributed web systems. In Proc. International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS'00). IEEE, San Francisco, CA, 20-27.
Pablo Castells, Saúl Vargas, and Jun Wang. 2011. Novelty and diversity metrics for recommender systems: Choice, discovery and relevance. In Proc. International Workshop on Diversity in Document Retrieval (DDR'11). Self-published. 29-36. http://www.dcs.gla.ac.uk/workshops/ddr2011/ddr2011.proceedings.pdf .
Kuang Chen, Harr Chen, Neil Conway, Joseph M. Hellerstein, and Tapan S. Parikh. 2011. Usher: Improving data quality with dynamic forms. IEEE Transactions on Knowledge and Data Engineering 23, 8 (2011), 1138-1153.
Tse-Hsun Chen, Weiyi Shang, Jinqiu Yang, Ahmed E. Hassan, Michael W. Godfrey, Mohamed Nasser, and Parminder Flora. 2016. An empirical study on the practice of maintaining object-relational mapping code in Java systems. In Proc. Working Conference on Mining Software Repositories (MSR'16). ACM, New York, NY, 165-176.
Takeshi Chusho, Katsuya Fujiwara, and Keiji Minamitani. 2002. Automatic filling in a form by an agent for web applications. In Proc. Asia-Pacific Software Engineering Conference (APSEC'02). IEEE, Berlin, Germany, 239-247.
Andy Cockburn and Carl Gutwin. 2009. A predictive model of human performance with scrolling and hierarchical lists. Human-Computer Interaction 24, 3 (2009), 273-314.
Ofer Dekel, Ohad Shamir, and Lin Xiao. 2010. Learning to classify with missing and corrupted features. Machine Learning 81, 2 (2010), 149-178.
Oscar Diaz, Itziar Otaduy, and Gorka Puente. 2013. User-driven automation of web form filling. In Proc. International Conference on Web Engineering (ICWE'13) (LNCS, Vol. 7977). Springer, Berlin, Germany, 171-185.
Khaled El Emam. 1999. Benchmarking Kappa: Interrater agreement in software process assessments. Empirical Software Engineering 4, 2 (1999), 113-133.
Wenfei Fan, Floris Geerts, and Xibei Jia. 2008. A revival of integrity constraints for data cleaning. Proc. VLDB Endowment' 08 1, 2 (2008), 1522-1523.
Sergio Firmenich, Vincent Gaits, Silvia Gordillo, Gustavo Rossi, and MarcoWinckler. 2012. Supporting users tasks with personal information management and web forms augmentation. In Proc. International Conference onWeb Engineering (ICWE'12) (LNCS, Vol. 7387). Springer, Oxford University Press, Berlin, Germany, 268-282.
Susan Fowler and Victor Stanwick. 2004. Web Application Design Handbook: Best Practices for Web-based Software. Morgan Kaufmann, Amsterdam, Boston, MA.
Nir Friedman, Dan Geiger, and Moises Goldszmidt. 1997. Bayesian network classifiers. Machine Learning 29, 2-3 (1997), 131-163.
Abdul Gafur. 2020. Updated tabular key and improved browser-based interactive key to species of Pratylenchus Filipjev (Nematoda: Pratylenchidae). Biodiversitas Journal of Biological Diversity 21, 8 (2020), 3780-3785.
José A. Gámez, Juan L. Mateo, and José M. Puerta. 2011. Learning Bayesian networks by hill climbing: Efficient methods based on progressive restriction of the neighborhood. Data Mining and Knowledge Discovery 22, 1-2 (2011), 106-148.
Mouzhi Ge, Carla Delgado-Battenfeld, and Dietmar Jannach. 2010. Beyond accuracy: Evaluating recommender systems by coverage and serendipity. In Proc. Recommender Systems (RecSys'10). ACM, New York, NY, 257-260.
Rafael S. Gonçalves, Martin J. O'Connor, Marcos Martínez-Romero, Attila L. Egyedi, Debra Willrett, John Graybeal, and Mark A. Musen. 2017. The CEDAR workbench: An ontology-assisted environment for authoring metadata that describe scientific experiments. In Proc. International Semantic Web Conference (ISWC'17) (LNCS, Vol. 10588). Springer International Publishing, Cham, 103-110.
Google. 2008. Chrome autofill forms. Retrieved February 18, 2020, from https://support.google.com/chrome.
Carl Gutwin and Andy Cockburn. 2006. Improving list revisitation with ListMaps. In Proc. Working Conference on Advanced Visual Interfaces (AVI'06). ACM, New York, NY, 396-403.
Melanie Hartmann and Max Muhlhauser. 2009. Context-aware form filling for web applications. In Proc. International Conference on Semantic Computing (ICSC'09). IEEE, Berkeley, CA, 221-228.
Bin He, Kevin Chen-Chuan Chang, and Jiawei Han. 2004. Discovering complexmatchings acrossweb query interfaces: A correlation mining approach. In Proc. International Conference on Knowledge Discovery and Data Mining (KDD'04). ACM, New York, NY, 148-157.
Carrie Heeter. 2000. Interactivity in the context of designed experiences. Journal of Interactive Advertising 1, 1 (2000), 3-14.
Jonathan L. Herlocker, Joseph A. Konstan, Loren G. Terveen, and John T. Riedl. 2004. Evaluating collaborative filtering recommender systems. ACM Transactions on Information Systems 22, 1 (2004), 5-53.
L. A. Hermens and J. C. Shlimmer. 1994. A machine-learning apprentice for the completion of repetitive forms. IEEE Expert 9, 1 (1994), 28-33.
Inma Hernández, Carlos R. Rivero, and David Ruiz. 2019. Deep web crawling: A survey.World Wide Web 22, 4 (2019), 1577-1610.
Jan Horsky, David R. Kaufman, Michael I. Oppenheim, and Vimla L. Patel. 2003. A framework for analyzing the cognitive complexity of computer-assisted clinical ordering. Journal of Biomedical Informatics 36, 1-2 (2003), 4-22.
Zhexue Huang. 1998. Extensions to the k-means algorithm for clustering large data sets with categorical values. Data Mining and Knowledge Discovery 2, 3 (1998), 283-304.
Caroline Jarrett and Gerry Gaffney. 2009. Forms That Work: Designing Web Forms for Usability. Morgan Kaufmann, Amsterdam, Boston, MA.
Emil Thorstensen Jensen, Martin Hansen, Evelyn Eika, and Frode Eika Sandnes. 2020. Country selection onweb forms: A comparison of dropdown menus, radio buttons and text field with autocomplete. In Proc. International Conference on Ubiquitous Information Management and Communication (IMCOM'20). IEEE, Taiwan, China, 1-4.
Xiao-Yuan Jing, Fumin Qi, Fei Wu, and Baowen Xu. 2016. Missing data imputation based on low-rank recovery and semi-supervised regression for software effort estimation. In Proc. International Conference on Software Engineering (ICSE'16). IEEE, Austin, TX, 607-618.
Chichang Jou. 2019. Schema extraction for deep web query interfaces using heuristics rules. Information Systems Frontiers 21, 1 (2019), 163-174.
Marius Kaminskas and Derek Bridge. 2016. Diversity, serendipity, novelty, and coverage: A survey and empirical analysis of beyond-accuracy objectives in recommender systems. ACM Transactions on Interactive Intelligent Systems 7, 1 (2016), 1-42.
Gustavo Zanini Kantorski, Viviane Pereira Moreira, and Carlos Alberto Heuser. 2015. Automatic filling of hidden web forms: A survey. ACM SIGMOD Record 44, 1 (2015), 24-35.
Mozhgan Karimi, Dietmar Jannach, and Michael Jugovac. 2018. News recommender systems-Survey and roads ahead. Information Processing & Management 54, 6 (2018), 1203-1227.
Reza Khajouei and M. W. M. Jaspers. 2010. The impact of CPOE medication systems' design aspects on usability, workflow and medication orders. Methods of Information in Medicine 49, 1 (2010), 03-19.
Trausti Kristjansson, Aron Culotta, Paul Viola, and Andrew McCallum. 2004. Interactive information extraction with constrained conditional random fields. In Proc. AAAI'04, Vol. 4. ACM, New York, NY, 412-418.
Matevẑ Kunaver and Tomaẑ Poẑrl. 2017. Diversity in recommender systems-A survey. Knowledge-based Systems 123 (2017), 154-162.
Hoyeop Lee, Jinbae Im, Seongwon Jang, Hyunsouk Cho, and Sehee Chung. 2019. Melu: Meta-learned user preference estimator for cold-start recommendation. In Proc. International Conference on Knowledge Discovery and Data Mining (KDD'19). ACM, New York, NY, 1073-1082.
Marcos Martínez-Romero, Martin J. O'Connor, Attila L. Egyedi, Debra Willrett, Josef Hardi, John Graybeal, and Mark A. Musen. 2019. Using association rule mining and ontologies to generate metadata recommendations from multiple biomedical databases. Database J. Biol. Databases Curation 2019 (2019), 25 pages.
S. McIntosh and Y. Kamei. 2017. Are fix-inducing changes a moving target? A longitudinal case study of just-in-time defect prediction. IEEE Transactions on Software Engineering 44, 5 (2017), 412-428.
Tim Menzies, Andrew Butcher, Andrian Marcus, Thomas Zimmermann, and David Cok. 2011. Local vs. global models for effort estimation and defect prediction. In Proc. International Conference on Automated Software Engineering (ASE'11). IEEE, Lawrence, KS, 343-351.
Microsoft. 2013. Change the default tab order for controls on a form. https://support.microsoft.com/en-us/office/ch ange-the-default-tab-order-for-controls-on-a-form-03d1599a-debf-4b66-a95b-e3e744210afe.
Kvanç Mulu, Yuriy Brun, and Alexandra Meliou. 2015. Preventing data errors with continuous testing. In Proc. ISSTA'15. ACM, New York, NY, 373-384.
John-Paul Ore, Sebastian Elbaum, Carrick Detweiler, and Lambros Karkazis. 2018. Assessing the type annotation burden. In Proc. International Conference on Automated Software Engineering (ASE'18). ACM, New York, NY, 190-201.
Spencer Pearson, José Campos, René Just, Gordon Fraser, Rui Abreu, Michael D. Ernst, Deric Pang, and Benjamin Keller. 2017. Evaluating and improving fault localization. In Proc. International Conference on Software Engineering (ICSE'17). IEEE, Buenos Aires, Argentina, 609-620.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12 (2011), 2825-2830.
Siyu Qian, Esther Munyisia, David Reid, David Hailey, Jade Pados, and Ping Yu. 2020. Trend in data errors after the implementation of an electronic medical record system: A longitudinal study in an Australian regional Drug and Alcohol Service. International Journal of Medical Informatics 144 (2020), 104292.
Adrian E. Raftery. 1995. Bayesian model selection in social research. Sociological Methodology 25 (1995), 111-163.
Martin Robillard, RobertWalker, and Thomas Zimmermann. 2009. Recommendation systems for software engineering. IEEE Software 27, 4 (2009), 80-86.
Enrico Rukzio, Chie Noda, Alexander De Luca, John Hamard, and Fatih Coskun. 2008. Automatic form filling on mobile devices. Pervasive and Mobile Computing 4, 2 (2008), 161-181.
Marton Sakal and Lazar Rakovic. 2012. Errors in building and using electronic tables: Financial consequences and minimisation techniques. Strategic Management 17, 3 (2012), 29-35.
Amr Rekaby Salama, Ozge Alaçam, and Wolfgang Menzel. 2018. Text completion using a context-integrating dependency parser. In Proc.Workshop on Representation Learning for NLP (RepL4NLP'18). ACL, Melbourne, Australia, 41-49.
Gunnar Schröder, Maik Thiele, and Wolfgang Lehner. 2011. Setting goals and choosing metrics for recommender system evaluations. In UCERSTI2 Workshop at the 5th ACM Conference on Recommender Systems (RecSys'11), Vol. 23. ACM, New York, NY, 53.
Andrew Sears and Ying Zha. 2003. Data entry for mobile devices using soft keyboards: Understanding the effects of keyboard size and user tasks. Journal of Human-Computer Interaction 16, 2 (2003), 163-184.
Guilherme A. Toda, Eli Cortez, Altigran S. da Silva, and Edleno de Moura. 2010. A probabilistic approach for automatically filling form-based web interfaces. Proc. of the VLDB Endowment 4, 3 (2010), 151-160.
Clare L. Tolley, Niamh E. Forde, Katherine L. Coffey, Dean F. Sittig, Joan S. Ash, Andrew K. Husband, DavidW. Bates, and Sarah P. Slight. 2018. Factors contributing to medication errors made when using computerized order entry in pediatrics: A systematic review. Journal of the American Medical Informatics Association 25, 5 (2018), 575-584.
Luigi Troiano, Cosimo Birtolo, and Roberto Armenise. 2017. Modeling and predicting the user next input by Bayesian reasoning. Soft Computing 21, 6 (2017), 1583-1600.
Qasim Umer, Hui Liu, and Inam Illahi. 2019. CNN-based automatic prioritization of bug reports. IEEE Transactions on Reliability 69, 4 (2019), 1341-1354.
Antal Van Den Bosch and Toine Bogers. 2008. Efficient context-sensitive word completion for mobile devices. In Proc. International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI'08). ACM, New York, NY, 465-470.
W3C School. 2021. HTML autocomplete Attribute. https://www.w3schools.com/tags/att_input_autocomple te.asp.
W3CSchools. 2017. HTML DOM input text object. https://www.w3schools.com/jsref/dom_obj_text.asp.
Shaohua Wang, Ying Zou, Iman Keivanloo, Bipin Upadhyaya, Joanna Ng, and Tinny Ng. 2014. Automatic reuse of user inputs to services among end-users in service composition. IEEE Transactions on Services Computing 8, 3 (2014), 343-355.
Shaohua Wang, Ying Zou, Joanna Ng, and Tinny Ng. 2017. Context-aware service input ranking by learning from historical information. IEEE Transactions on Services Computing 14, 1 (2017), 97-110.
Johanna I. Westbrook, Melissa T. Baysari, Ling Li, Rosemary Burke, Katrina L. Richardson, and Richard O. Day. 2013. The safety of electronic prescribing: Manifestations, mechanisms, and rates of system-related errors associated with two commercial systems in hospitals. Journal of the American Medical Informatics Association 20, 6 (2013), 1159-1167.
Marco Winckler, Vicent Gaits, Dong-Bach Vo, Firmenich Sergio, and Gustavo Rossi. 2011. An approach and tool support for assisting users to fill-in web forms with personal information. In Proc. International Conference on Design of Communication (SIGDOC'11). ACM, New York, NY, 195-202.
Xinli Yang, David Lo, Xin Xia, Lingfeng Bao, and Jianling Sun. 2016. Combining word embedding with information retrieval to recommend similar bug reports. In Proc. International Symposium on Software Reliability Engineering (ISSRE' 16). IEEE, Ottawa, ON, Canada, 127-137.
Xin Ye, Razvan Bunescu, and Chang Liu. 2014. Learning to rank relevant files for bug reports using domain knowledge. In Proc. International Symposium on Foundations of Software Engineering (FSE'14). ACM, New York, NY, 689-699.
Mingrui Ray Zhang, Shumin Zhai, and Jacob O.Wobbrock. 2019. Text entry throughput: Towards unifying speed and accuracy in a single performance metric. In Proc. CHI Conference on Human Factors in Computing Systems (CHI'19). ACM, New York, NY, 1-13.
