Impact of Design Decisions in Scanpath Modeling

Eye Tracking; Interaction Design; Visual Saliency; Computational modelling; Deep learning; Design parameters; Scan path; User interface designs; Human-Computer Interaction; Computer Science - Computer Vision and Pattern Recognition

Abstract :

[en] Modeling visual saliency in graphical user interfaces (GUIs) allows to understand how people perceive GUI designs and what elements attract their attention. One aspect that is often overlooked is the fact that computational models depend on a series of design parameters that are not straightforward to decide. We systematically analyze how different design parameters affect scanpath evaluation metrics using a state-of-the-art computational model (DeepGaze++). We particularly focus on three design parameters: input image size, inhibition-of-return decay, and masking radius. We show that even small variations of these design parameters have a noticeable impact on standard evaluation metrics such as DTW or Eyenalysis. These effects also occur in other scanpath models, such as UMSS and ScanGAN, and in other datasets such as MASSVIS. Taken together, our results put forward the impact of design decisions for predicting users’ viewing behavior on GUIs.

Research center :

ULHPC - University of Luxembourg: High Performance Computing

Disciplines :

Computer science

Author, co-author :

EMAMI, Parvin ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)

Jiang, Yue ; Aalto University, Finland

Guo, Zixin ; Aalto University, Finland

LEIVA, Luis A. ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)

External co-authors :

yes

Language :

English

Title :

Impact of Design Decisions in Scanpath Modeling

Publication date :

28 May 2024

Journal title :

Proceedings of the ACM on Human-Computer Interaction

eISSN :

2573-0142

Publisher :

Association for Computing Machinery

Volume :

Issue :

ETRA

Pages :

1-16

Peer reviewed :

Peer Reviewed verified by ORBi

Additional URL :

https://dl.acm.org/doi/pdf/10.1145/3655602

European Projects :

HE - 101071147 - SYMBIOTIK - Context-aware adaptive visualizations for critical decision making

Funders :

UE - Union Européenne

Funding text :

Research supported by the Horizon 2020 FET program of the European Union through the ERA-NET Cofund funding (grant CHIST-ERA-20-BCI-001) and the Pathfinder program of the European Innovation (SYMBIOTIK project, grant 101071147).

Commentary :

16 pages

Available on ORBilu :

since 25 January 2025

Statistics

Number of views

86 (15 by Unilu)

Number of downloads

30 (2 by Unilu)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

OpenAlex citations

Bibliography

Nicola C Anderson, Fraser Anderson, Alan Kingstone, and Walter F Bischof. 2015. A comparison of scanpath comparison methods. Behavior research methods 47 (2015), 1377–1392.
Marc Assens, Xavier Giro-i Nieto, Kevin McGuinness, and Noel E O’Connor. 2018. PathGAN: Visual scanpath prediction with generative adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops. 0–0.
Wentao Bao and Zhenzhong Chen. 2020. Human scanpath prediction based on deep convolutional saccadic model. Neurocomputing 404 (2020), 154–164.
Donald J Berndt and James Clifford. 1994. Using dynamic time warping to find patterns in time series. In Proceedings of the 3rd international conference on knowledge discovery and data mining. 359–370.
Michelle A. Borkin, Zoya Bylinskii, Nam Wook Kim, Constance May Bainbridge, Chelsea S. Yeh, Daniel Borkin, Hanspeter Pfister, and Aude Oliva. 2016. Beyond Memorability: Visualization Recognition and Recall. IEEE Transactions on Visualization and Computer Graphics 22, 1 (2016).
Xianyu Chen, Ming Jiang, and Qi Zhao. 2021. Predicting human scanpaths in visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10876–10885.
Zhenzhong Chen and Wanjie Sun. 2018. Scanpath Prediction for Visual Attention using IOR-ROI LSTM.. In IJCAI. 642–648.
Aladine Chetouani and Leida Li. 2020. On the use of a scanpath predictor and convolutional neural network for blind image quality assessment. Signal Processing: Image Communication 89 (2020), 115963.
Marcella Cornia, Lorenzo Baraldi, Giuseppe Serra, and Rita Cucchiara. 2018. SAM: Pushing the limits of saliency prediction models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 1890–1892.
Erwan Joel David, Pierre Lebranchu, Matthieu Perreira Da Silva, and Patrick Le Callet. 2019. Predicting artificial visual field losses: A gaze-based inference study. Journal of Vision 19, 14 (2019), 22–22.
Ryan Anthony Jalova de Belen, Tomasz Bednarz, and Arcot Sowmya. 2022. Scanpathnet: A recurrent mixture density network for scanpath prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5010–5020.
Sukru Eraslan, Yeliz Yesilada, and Simon Harper. 2016. Eye tracking scanpath analysis on web pages: how many users?. In Proceedings of the ninth biennial ACM symposium on eye tracking research & applications. 103–110.
Ramin Fahimi. 2018. Sequential selection, saliency and scanpaths. (2018).
Ramin Fahimi and Neil DB Bruce. 2021. On metrics for measuring scanpath similarity. Behavior Research Methods 53 (2021), 609–628.
Camilo Fosco, Vincent Casser, Amish Kumar Bedi, Peter O’Donovan, Aaron Hertzmann, and Zoya Bylinskii. 2020. Predicting visual importance across graphic design types. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. 249–260.
Laurent Itti, Christof Koch, and Ernst Niebur. 1998. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on pattern analysis and machine intelligence 20, 11 (1998), 1254–1259.
Yue Jiang, Luis A Leiva, Hamed Rezazadegan Tavakoli, Paul RB Houssel, Julia Kylmälä, and Antti Oulasvirta. 2023. UEyes: Understanding Visual Saliency across User Interface Types. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–21.
Raymond M Klein. 2000. Inhibition of return. Trends in cognitive sciences 4, 4 (2000), 138–147.
Matthias Kümmerer and Matthias Bethge. 2021. State-of-the-art in human scanpath prediction. arXiv preprint arXiv:2102.12239 (2021).
Matthias Kümmerer, Matthias Bethge, and Thomas SA Wallis. 2022. DeepGaze III: Modeling free-viewing human scanpaths with deep learning. Journal of Vision 22, 5 (2022), 7–7.
Daniël Lakens. 2013. Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Front. Psychol. 4, 863 (2013).
Luis A. Leiva, Yunfei Xue, Avya Bansal, Hamed R. Tavakoli, Tuğçe Köroğlu, Jingzhou Du, Niraj R. Dayama, and Antti Oulasvirta. 2020. Understanding Visual Saliency in Mobile User Interfaces. In Proceedings of the Intl. Conf. on Human-computer interaction with mobile devices and services (MobileHCI).
Mu Li, Kanglong Fan, and Kede Ma. 2023. Scanpath Prediction in Panoramic Videos via Expected Code Length Minimization. arXiv preprint arXiv:2305.02536 (2023).
Yue Li, Dong Liu, Houqiang Li, Li Li, Zhu Li, and Feng Wu. 2018. Learning a convolutional neural network for image compact-resolution. IEEE Transactions on Image Processing 28, 3 (2018), 1092–1107.
Daniel Martin, Ana Serrano, Alexander W Bergman, Gordon Wetzstein, and Belen Masia. 2022. ScanGAN360: A generative model of realistic scanpaths for 360 images. IEEE Transactions on Visualization and Computer Graphics 28, 5 (2022), 2003–2013.
Sebastiaan Mathôt, Filipe Cristino, Iain D Gilchrist, and Jan Theeuwes. 2012. A simple way to estimate similarity between pairs of eye movement sequences. Journal of Eye Movement Research 5, 1 (2012), 1–15.
Meinard Müller. 2007. Dynamic time warping. Information retrieval for music and motion (2007), 69–84.
Thuyen Ngo and BS Manjunath. 2017. Saccade gaze prediction using a recurrent neural network. In 2017 IEEE International Conference on Image Processing (ICIP). IEEE, 3435–3439.
Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. 2022. On aliased resizing and surprising subtleties in GAN evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11410–11420.
Michael I Posner, Yoav Cohen, et al. 1984. Components of visual orienting. Attention and performance X: Control of language processes 32 (1984), 531–556.
Xiangjie Sui, Yuming Fang, Hanwei Zhu, Shiqi Wang, and Zhou Wang. 2023. ScanDMM: A Deep Markov Model of Scanpath Prediction for 360° Images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 6989–6999.
Wanjie Sun, Zhenzhong Chen, and Feng Wu. 2019. Visual scanpath prediction using IOR-ROI recurrent mixture density network. IEEE transactions on pattern analysis and machine intelligence 43, 6 (2019), 2101–2118.
Nađa Terzimehić, Renate Häuslschmid, Heinrich Hussmann, and MC Schraefel. 2019. A review & analysis of mindfulness research in HCI: Framing current lines of research and future opportunities. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–13.
Xiaowei Wang, Xubo Li, Haiying Wang, Wenning Zhao, and Xia Liu. 2023. An Improved Dynamic Time Warping Method Combining Distance Density Clustering for Eye Movement Analysis. Journal of Mechanics in Medicine and Biology 23, 02 (2023), 2350031.
Yao Wang, Andreas Bulling, et al. 2023. Scanpath prediction on information visualisations. IEEE Transactions on Visualization and Computer Graphics (2023).
Calden Wloka, Iuliia Kotseruba, and John K Tsotsos. 2018. Active fixation control to predict saccade sequences. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3184–3193.
Chen Xia, Junwei Han, Fei Qi, and Guangming Shi. 2019. Predicting human saccadic scanpaths based on iterative representation learning. IEEE Transactions on Image Processing 28, 7 (2019), 3502–3515.
Mulong Xie, Sidong Feng, Zhenchang Xing, Jieshan Chen, and Chunyang Chen. 2020. UIED: a hybrid tool for GUI element detection. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1655–1659.
Fei Yan, Cheng Chen, Peng Xiao, Siyu Qi, Zhiliang Wang, and Ruoxiu Xiao. 2021. Review of visual saliency prediction: Development process from neurobiological basis to deep models. Applied Sciences 12, 1 (2021), 309.
Jiawei Yang, Guangtao Zhai, and Huiyu Duan. 2019. Predicting the visual saliency of the people with VIMS. In 2019 IEEE Visual Communications and Image Processing (VCIP). IEEE, 1–4.
Yucheng Zhu, Guangtao Zhai, Xiongkuo Min, and Jiantao Zhou. 2019. The prediction of saliency map for head and eye movements in 360 degree images. IEEE Transactions on Multimedia 22, 9 (2019), 2331–2344.
Yucheng Zhu, Guangtao Zhai, Yiwei Yang, Huiyu Duan, Xiongkuo Min, and Xiaokang Yang. 2021. Viewing behavior supported visual saliency predictor for 360 degree videos. IEEE Transactions on Circuits and Systems for Video Technology 32, 7 (2021), 4188–4201.