Modeling how menu search strategies develop with experience

Computational models; Decision making; Learning; Linear menus; Visual search; Arbitration mechanisms; Computational modelling; Decisions makings; Linear menu; Predictive models; Search behavior; Search strategies; Visual search strategies; Human Factors and Ergonomics; Software; Education; Engineering (all); Human-Computer Interaction; Hardware and Architecture

Abstract :

[en] To find an item in a menu, users can follow different visual search strategies, such as scanning items one by one (serial search) or trying to remember where the item was (recall search). However, building predictive models of search behavior has turned out to be challenging, because these strategies evolve with practice. To address this challenge, we study theory-inspired models of visual search in linear menus and propose a novel arbitration mechanism to coordinate the adoption of such visual search strategies. Given a menu design and the user's previous experience with it, our approach predicts when different search strategies (serial, recall, random) will be adopted and which menu item will be fixated next. Our results (1) describe empirical data plausibly with psychologically valid and interpretable models, (2) provide new insights about how search strategies evolve with practice, and (3) show how to infer search strategy from eye tracking data. To sum up, the models provide a foundation to better understand how users learn to scan linear menus.

Disciplines :

Computer science

Author, co-author :

Bailly, Gilles ; Sorbonne Université, CNRS, ISIR, France

Duarte, Daniel; University of Luxembourg, Luxembourg

Oulasvirta, Antti ; Aalto University, Finland

LEIVA, Luis A. ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)

External co-authors :

yes

Language :

English

Title :

Modeling how menu search strategies develop with experience

Publication date :

November 2025

Journal title :

International Journal of Human-Computer Studies

ISSN :

1071-5819

eISSN :

1095-9300

Publisher :

Academic Press

Volume :

205

Pages :

103615

Peer reviewed :

Peer Reviewed verified by ORBi

Focus Area :

Computational Sciences

Additional URL :

https://api.elsevier.com/content/article/PII:S1071581925001727?httpAccept=text/xml

European Projects :

HE - 101071147 - SYMBIOTIK - Context-aware adaptive visualizations for critical decision making

FnR Project :

FNR15722813 - BANANA - Brainsourcing For Affective Attention Estimation, 2021 (01/02/2022-31/01/2025) - Luis Leiva

Funders :

European Union

Funding text :

We thank Benoit Girard for his useful feedak. This work was supported by the Department of Information and Communications Engineering - Aalto University , the Finnish Center for Artificial Intelligence (FCAI) , the ANR NeuroHCI project ( ANR-22-CE33-0006-01 ), the Academy of Finland through the Subjective Functions (grant 357578 ), the Horizon 2020 FET program of the European Union (grant CHIST-ERA-20-BCI-001 ) and the European Innovation Council Pathfinder program (grant 101071147 ).

Available on ORBilu :

since 22 September 2025

Statistics

Number of views

50 (7 by Unilu)

Number of downloads

50 (0 by Unilu)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

OpenAlex citations

Bibliography

Anderson, J.R., Lebiere, C.J., The Atomic Components of Thought. 2014, Psychology Press, 10.4324/9781315805696.
Bailly, G., Khamassi, M., Girard, B., Computational model of the transition from novice to expert interaction techniques. ACM Trans. Comput. Hum. Interact., 2022, 10.1145/3505557 Just Accepted.
Bailly, G., Lecolinet, E., Nigay, L., Visual menu techniques. ACM Comput. Surv., 49(4), 2016, 10.1145/3002171.
Bailly, G., Malacria, S., Command selection. Vanderdonckt, J., Palanque, P., Winckler, M., (eds.) Handbook of Human Computer Interaction, 2022, Springer International Publishing, Cham, 1–35, 10.1007/978-3-319-27648-9_19-1.
Bailly, G., Oulasvirta, A., Brumby, D.P., Howes, A., Model of visual search and selection time in linear menus. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’14, 2014, ACM, New York, NY, USA, 3865–3874, 10.1145/2556288.2557093.
Bailly, G., Oulasvirta, A., Kötzing, T., Hoppe, S., MenuOptimizer: Interactive optimization of menu systems. Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology, UIST ’13, 2013, Association for Computing Machinery, New York, NY, USA, 331–342, 10.1145/2501988.2502024.
Byrne, M.D., Anderson, J.R., Douglass, S., Matessa, M., Eye tracking the visual search of click-down menus. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’99, 1999, Association for Computing Machinery, New York, NY, USA, 402–409, 10.1145/302979.303118.
Chen, X., Bailly, G., Brumby, D.P., Oulasvirta, A., Howes, A., The emergence of interactive behavior: A model of rational menu search. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI ’15, 2015, Association for Computing Machinery, New York, NY, USA, 4217–4226, 10.1145/2702123.2702483.
Cinotti, F., Fresno, V., Aklil, N., Coutureau, E., Girard, B., Marchand, A.R., Khamassi, M., Dopamine blockade impairs the exploration-exploitation trade-off in rats. Sci. Rep. 9:1 (2019), 1–14.
Clarke, A.D., Stainer, M.J., Tatler, B.W., Hunt, A.R., The saccadic flow baseline: Accounting for image-independent biases in fixation behavior. J. Vis., 17(11), 2017, 12.
Cockburn, A., Gutwin, C., A predictive model of human performance with scrolling and hierarchical lists. Hum. Comput. Interact. 24:3 (2009), 273–314, 10.1080/07370020902990402 arXiv:https://www.tandfonline.com/doi/pdf/10.1080/07370020902990402.
Cockburn, A., Gutwin, C., Greenberg, S., A predictive model of menu performance. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’07, 2007, ACM, New York, NY, USA, 627–636, 10.1145/1240624.1240723.
Coutrot, A., Hsiao, J.H., Chan, A.B., Scanpath modeling and classification with hidden Markov models. Behav. Res. Methods 50:1 (2018), 362–379.
Cox, A.L., Silva, M.M., 2006. The role of mouse movements in interactive search. In: Proceedings of the Annual Meeting of the Cognitive Science Society. Vol. 28.
Daw, N., Trial-by-trial data analysis using computational models. Decision Making, Affect, and Learning, 2011, Oxford University Press, 10.1093/acprof:oso/9780199600434.003.0001 Publisher Copyright: © The International Association for the study of Attention and Performance, 2011. All rights reserved.
Dayama, N.R., Shiripour, M., Oulasvirta, A., Ivanko, E., Karrenbauer, A., Foraging-based optimization of menu systems. Int. J. Hum.-Comput. Stud., 151, 2021, 102624, 10.1016/j.ijhcs.2021.102624 URL: https://www.sciencedirect.com/science/article/pii/S1071581921000422.
Delamare, W., Neshati, A., Irani, P., Ren, X., An analytic model for time efficient personal hierarchies. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI ’19, 2019, Association for Computing Machinery, New York, NY, USA, 1–11, 10.1145/3290605.3300598.
Fahimi, R., Bruce, N.D., On metrics for measuring scanpath similarity. Behav. Res. Methods 53 (2021), 609–628.
Findlater, L., Moffatt, K., McGrenere, J., Dawson, J., Ephemeral adaptation: The use of gradual onset to improve menu selection performance. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’09, 2009, Association for Computing Machinery, New York, NY, USA, 1655–1664, 10.1145/1518701.1518956.
Findlay, J.M., Gilchrist, I.D., Active vision: The psychology of looking and seeing, 2003, Oxford University Press.
Fitts, P.M., Posner, M.I., Human performance. 1967, Brooks/Cole.
Gershman, S.J., Origin of perseveration in the trade-off between reward and complexity. Cogn., 204, 2020, 104394.
Goubko, M.V., Danilenko, A.I., 2010. An automated routine for menu structure optimization. In: Proceedings of the 2nd ACM SIGCHI Symposium on Engineering Interactive Computing Systems. pp. 67–76.
Grossman, T., Dragicevic, P., Balakrishnan, R., Strategies for accelerating on-line learning of hotkeys. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’07, 2007, ACM, New York, NY, USA, 1591–1600, 10.1145/1240624.1240865 URL: http://doi.acm.org/10.1145/1240624.1240865.
Gutwin, C., Cockburn, A., Scarr, J., Malacria, S., Olson, S.C., Faster command selection on tablets with FastTap. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’14, 2014, Association for Computing Machinery, New York, NY, USA, 2617–2626, 10.1145/2556288.2557136.
Halverson, T., Hornof, A.J., A computational model of “Active vision” for visual search in human–Computer interaction. Hum. Comput. Interact. 26:4 (2011), 285–314, 10.1080/07370024.2011.625237 arXiv:https://doi.org/10.1080/07370024.2011.625237.
Henson, D., Corrective saccades: Effects of altering visual feedback. Vis. Res. 18:1 (1978), 63–67, 10.1016/0042-6989(78)90078-0 URL: https://www.sciencedirect.com/science/article/pii/0042698978900780.
Hornof, A.J., Cognitive strategies for the visual search of hierarchical computer displays. Hum. Comput. Interact. 19:3 (2004), 183–223, 10.1207/s15327051hci1903_1.
Hornof, A.J., Kieras, D.E., Cognitive modeling reveals menu search in both random and systematic. Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, CHI ’97, 1997, Association for Computing Machinery, New York, NY, USA, 107–114, 10.1145/258549.258621.
Hu, Z., Bulling, A., Li, S., Wang, G., Fixationnet: Forecasting eye fixations in task-oriented virtual environments. IEEE Trans. Vis. Comput. Graphics 27:5 (2021), 2681–2690.
Jacobs, A.M., Grainger, J., Models of visual word recognition: sampling the state of the art. J. Exp. Psychol. [Hum. Percept.], 20(6), 1994, 1311.
Jiang, Y., Guo, Z., Rezazadegan Tavakoli, H., Leiva, L.A., Oulasvirta, A., 2024. EyeFormer: predicting personalized scanpaths with transformer-guided reinforcement learning. In: Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology. pp. 1–15.
Jokinen, J.P., Wang, Z., Sarcar, S., Oulasvirta, A., Ren, X., Adaptive feature guidance: Modelling visual search with graphical layouts. Int. J. Hum.-Comput. Stud., 136, 2020, 102376, 10.1016/j.ijhcs.2019.102376 URL: https://www.sciencedirect.com/science/article/pii/S1071581919301429.
Kangasrääsiö, A., Athukorala, K., Howes, A., Corander, J., Kaski, S., Oulasvirta, A., Inferring cognitive models from data using approximate Bayesian computation. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI ’17, 2017, Association for Computing Machinery, New York, NY, USA, 1295–1306, 10.1145/3025453.3025576.
Klein, R.M., Inhibition of return. Trends Cogn. Sci. 4:4 (2000), 138–147, 10.1016/S1364-6613(00)01452-2 URL: https://www.sciencedirect.com/science/article/pii/S1364661300014522.
Kümmerer, M., Bethge, M., State-of-the-art in human scanpath prediction. 2021 arXiv preprint arXiv:2102.12239.
Kümmerer, M., Wallis, T.S., Bethge, M., Information-theoretic model comparison unifies saliency metrics. Proc. Natl. Acad. Sci. 112:52 (2015), 16054–16059.
Kunar, M.A., Flusberg, S., Wolfe, J.M., The role of memory and restricted context in repeated visual search. Percept. Psychophys. 70:2 (2008), 314–328.
Lane, D.M., Napier, H.A., Peres, S.C., Sandor, A., Hidden Costs of Graphical User Interfaces: Failure to Make the Transition from Menus and Icon Toolbars to Keyboard Shortcuts. Int. J. Hum.-Comput. Interact. 18:2 (2005), 133–144, 10.1207/s15327590ijhc1802_1 URL: http://www.tandfonline.com/doi/abs/10.1207/s15327590ijhc1802_1.
Lee, N., Interview with bill kinder: January 13, 2005. Comput. Entertain., 3(1), 2005, 10.1145/1057270.1057278 URL: http://doi.acm.org/10.1145/1057270.1057278.
Lee, E., MacGregor, J., Minimizing user search time in menu retrieval systems. Hum. Factors 27:2 (1985), 157–162.
Lewis, R.L., Howes, A., Singh, S., Computational rationality: Linking mechanism and behavior through bounded utility maximization. Top. Cogn. Sci. 6:2 (2014), 279–311.
Li, Y., Bengio, S., Bailly, G., Predicting human performance in vertical menu selection using deep learning. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, 2018, Association for Computing Machinery, New York, NY, USA, 1–7, 10.1145/3173574.3173603.
Li, Z., Ko, Y.-J., Putkonen, A., Feiz, S., Ashok, V., Ramakrishnan, I., Oulasvirta, A., Bi, X., Modeling touch-based menu selection performance of blind users via reinforcement learning. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, 2023, Association for Computing Machinery, New York, NY, USA, 10.1145/3544548.3580640.
Matsui, S., Yamada, S., Genetic algorithm can optimize hierarchical menus. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’08, 2008, Association for Computing Machinery, New York, NY, USA, 1385–1388, 10.1145/1357054.1357271.
McDonald, J.E., Stone, J.D., Liebelt, L.S., Searching for items in menus: The effects of organization and type of target. Proceedings of the Human Factors Society Annual Meeting, Vol. 27, 1983, SAGE Publications Sage CA, Los Angeles, CA, 834–837.
Molinaro, G., Collins, A.G., A goal-centric outlook on learning. Trends Cogn. Sci., 2023.
Neisser, U., Beller, H.K., Searching through word lists. Br. J. Psychol. 56:4 (1965), 349–358.
Newell, A., Rosenbloom, P.S., Mechanisms of Skill Acquisition and the Law of Practice. 1993, MIT Press, Cambridge, MA, USA, 81–135.
Nilsen, E.L., Perceptual-Motor Control in Human-Computer Interaction. 1991, University of Michigan.
Norman, K.L., The Psychology of Menu Selection: Designing Cognitive Control at the Human/computer Interface. 1991, Intellect Books.
Price, K.V., Storn, R.M., Lampinen, J.A., Differential Evolution: A Practical Approach to Global Optimization. first ed., 2005, Springer Berlin, Heidelberg.
Raftery, A.E., Bayesian model selection in social research. Sociol. Methodol., 1995, 111–163.
Rayner, K., Eye movements in reading and information processing: 20 years of research. Psychol. Bull., 124(3), 1998, 372.
Schütt, H.H., Rothkegel, L.O., Trukenbrod, H.A., Reich, S., Wichmann, F.A., Engbert, R., Likelihood-based parameter estimation and comparison of dynamical cognitive models. Psychol Rev, 124(4), 2017, 505.
Sears, A., Shneiderman, B., Split menus: Effectively using selection frequency to organize menus. ACM Trans. Comput. Hum. Interact. 1:1 (1994), 27–51, 10.1145/174630.174632.
Somberg, B.L., A comparison of rule-based and positionally constant arrangements of computer menu items. Proceedings of the SIGCHI/GI Conference on Human Factors in Computing Systems and Graphics Interface, CHI ’87, 1986, Association for Computing Machinery, New York, NY, USA, 255–260, 10.1145/29933.275639.
Sourulahti, S., Janssen, C.P., Jokinen, J.P., Modeling rational adaptation of visual search to hierarchical structures. 2024 arXiv:2409.08967. URL: https://arxiv.org/abs/2409.08967.
Tak, S., Westendorp, P., van Rooij, I., Satisficing and the use of keyboard shortcuts: Being good enough is enough?. Interact. Comput. 25:5 (2013), 404–416, 10.1093/iwc/iwt016.
Todi, K., Bailly, G., Leiva, L.A., Oulasvirta, A., Adapting user interfaces with model-based reinforcement learning. 2021 arXiv preprint arXiv:2103.06807.
Tsandilas, T., schraefel, m.c., Bubbling menus: A selective mechanism for accessing hierarchical drop-down menus. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’07, 2007, Association for Computing Machinery, New York, NY, USA, 1195–1204, 10.1145/1240624.1240806.
Viejo, G., Khamassi, M., Brovelli, A., Girard, B., Modeling choice and reaction time during arbitrary visuomotor learning through the coordination of adaptive working memory and reinforcement learning. Front. Behav. Neurosci., 9, 2015, 225.
Wang, Y., Bâce, M., Bulling, A., Scanpath prediction on information visualisations. IEEE Trans. Vis. Comput. Graphics 30:7 (2023), 3902–3914, 10.1109/TVCG.2023.3242293.
Wilson, R.C., Collins, A.G., Ten simple rules for the computational modeling of behavioral data. Elife, 8, 2019, e49547.
Wolfe, J.M., Klempen, N., Dahlen, K., Postattentive vision. J. Exp. Psychol. [Hum. Percept.], 26(2), 2000, 693.