Affective computing; Affective feedback; Emotion detection; Affective state; Emotional response; Evaluation methods; Functional near infrared spectroscopy; Human brain; Ranking methods; Information Systems
Résumé :
[en] Information retrieval (IR) relies on a general notion of relevance, which is used as the principal foundation for ranking and evaluation methods. However, IR does not account for more a nuanced affective experience. Here, we consider the emotional response decoded directly from the human brain as an alternative dimension of relevance. We report an experiment covering seven different scenarios in which we measure and predict how users emotionally respond to visual image contents by using functional near-infrared spectroscopy (fNIRS) neuroimaging on two commonly used affective dimensions: valence (negativity and positivity) and arousal (bored-ness and excitedness). Our results show that affective states can be successfully decoded using fNIRS, and utilized to complement the present notion of relevance in IR studies. For example, we achieved 0.39 Balanced accuracy and 0.61 AUC in 4-class classification of affective states (vs. 0.25 Balanced accuracy and 0.5 AUC of a random classifier). Likewise, we achieved 0.684 Precision@20 when retrieving high-arousal images. Our work opens new avenues for incorporating emotional states in IR evaluation, affective feedback, and information filtering.
Disciplines :
Sciences informatiques
Auteur, co-auteur :
Ruotsalo, Tuukka ; University of Copenhagen, Denmark ; University of Helsinki, Finland
Spapé, Michiel M. ; University of Helsinki, Finland
Mäkelä, Kalle ; University of Helsinki, Finland
LEIVA, Luis A. ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)
Co-auteurs externes :
yes
Langue du document :
Anglais
Titre :
Affective Relevance Inferring Emotional Responses via fNIRS Neuroimaging
Date de publication/diffusion :
19 juillet 2023
Nom de la manifestation :
Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval
Organisateur de la manifestation :
ACM
Lieu de la manifestation :
Taipei, Twn
Date de la manifestation :
23-07-2023 => 27-07-2023
Manifestation à portée :
International
Titre de l'ouvrage principal :
SIGIR 2023 - Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval
HE - 101071147 - SYMBIOTIK - Context-aware adaptive visualizations for critical decision making
Projet FnR :
FNR15722813 - Brainsourcing For Affective Attention Estimation, 2021 (01/02/2022-31/01/2025) - Luis Leiva
Organisme subsidiant :
Union Européenne
Subventionnement (détails) :
This work is supported by the Academy of Finland (grants 352915, 350323, 336085, 322653), the Horizon 2020 FET program of the European Union (BANANA, grant CHIST-ERA-20-BCI-001), and the EIC Pathfinder program (SYMBIOTIK, grant 101071147).
Eugene Agichtein, Eric Brill, and Susan Dumais. 2006. Improving Web Search Ranking by Incorporating User Behavior Information. In Proc. SIGIR.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework. In Proc. KDD.
Ioannis Arapakis, Joemon M Jose, and Philip D Gray. 2008. Affective feedback: an investigation into the role of emotions in the information seeking process. In Proc. SIGIR.
Ioannis Arapakis and Luis A. Leiva. 2020. Learning Efficient Representations of Mouse Movements to Predict User Attention. In Proc. SIGIR.
Pedro Avero and Manuel G Calvo. 2006. Affective priming with pictures of emotional scenes: The role of perceptual similarity and category relatedness. Span. J. Psychol. 9, 1 (2006).
Hasan Ayaz, Meltem Izzetoglu, Kurtulus Izzetoglu, and Banu Onaral. 2019. The use of functional near-infrared spectroscopy in neuroergonomics. In Neuroergonomics. Elsevier.
Michela Balconi, Elisabetta Grippa, and Maria Elide Vanutelli. 2015. Resting lateralized activity predicts the cortical response and appraisal of emotions: an fNIRS study. Soc. Cogn. Affect. Neurosci. 10, 12 (2015).
Michela Balconi, Elisabetta Grippa, and Maria Elide Vanutelli. 2015. What hemodynamic (fNIRS), electrophysiological (EEG) and autonomic integrated measures can tell us about emotional processing. Brain Cogn. 95 (2015).
Oswald Barral, Manuel JA Eugster, Tuukka Ruotsalo, Michiel M Spapé, Ilkka Kosunen, Niklas Ravaja, Samuel Kaski, and Giulio Jacucci. 2015. Exploring peripheral physiology as a predictor of perceived relevance in information retrieval. In Proc. IUI.
Lukas Brückner, Ioannis Arapakis, and Luis A. Leiva. 2021. When Choice Happens: A Systematic Examination of Mouse Movement Length for Decision Making in Web Search. In Proc. SIGIR.
Scott C Bunce, Meltem Izzetoglu, Kurtulus Izzetoglu, Banu Onaral, and Kambiz Pourrezaei. 2006. Functional near-infrared spectroscopy. IEEE Eng. Med. Biol. Mag. 25, 4 (2006).
Luis Carretié. 2014. Exogenous (automatic) attention to emotional stimuli: a review. Cogn. Affect. Behav. Neurosci. 14, 4 (2014).
Ming Chen, Lu Zhang, and Jan P Allebach. 2015. Learning deep features for image emotion classification. In Proc. ICIP.
Xu Cui, Signe Bray, and Allan L Reiss. 2010. Functional near infrared spectroscopy (NIRS) signal improvement based on negative correlation between oxygenated and deoxygenated hemoglobin dynamics. Neuroimage 49, 4 (2010), 3039-3046.
Keith M Davis, Carlos de la Torre-Ortiz, and Tuukka Ruotsalo. 2022. Brain-Supervised Image Editing. In Proc. CVPR.
Keith M. Davis, Michiel Spapé, and Tuukka Ruotsalo. 2022. Contradicted by the Brain: Predicting Individual and Group Preferences via Brain-Computer Interfacing. IEEE Trans. Affect. Comput. (2022).
Keith M. Davis III, Michiel Spapé, and Tuukka Ruotsalo. 2021. Collaborative Filtering with Preferences Inferred from Brain Signals. In Proc. WWW.
Carlos de la Torre-Ortiz, Michiel M Spapé, Lauri Kangassalo, and Tuukka Ruotsalo. 2020. Brain relevance feedback for interactive image generation. In Proc. UIST.
David T Delpy, Mark Cope, Pieter van der Zee, Simon Arridge, Susan Wray, and JS Wyatt. 1988. Estimation of optical pathlength through tissue from direct time of flight measurement. Phys. Med. Biol. 33, 12 (1988).
Manuel JA Eugster, Tuukka Ruotsalo, Michiel M Spapé, Oswald Barral, Niklas Ravaja, Giulio Jacucci, and Samuel Kaski. 2016. Natural brain-information interfaces: Recommending information by relevance inferred from human brain signals. Sci. Rep. 6, 1 (2016).
Manuel J.A. Eugster, Tuukka Ruotsalo, Michiel M. Spapé, Ilkka Kosunen, Oswald Barral, Niklas Ravaja, Giulio Jacucci, and Samuel Kaski. 2014. Predicting Term-Relevance from Brain Signals. In Proc. SIGIR.
Frank A Fishburn, Ruth S Ludlum, Chandan J Vaidya, and Andrei V Medvedev. 2019. Temporal derivative distribution repair (TDDR): a motion correction method for fNIRS. Neuroimage 184 (2019).
Don C Fowles. 1980. The three arousal model: Implications of Gray's two-factor learning theory for heart rate, electrodermal activity, and psychopathy. Int. J. Psychophysiol. 17, 2 (1980), 87-104.
E Grippa, Maria Elide Vanutelli, I Venturella, E Molteni, and Michela Balconi. 2014. Hemodynamic responses (fNIRS) and EEG modulation of prefrontal cortex during emotion processing. In Proc. SIPF.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In Proc. CVPR.
Fabian Herold, Patrick Wiegel, Felix Scholkmann, and Notger G Müller. 2018. Applications of functional near-infrared spectroscopy (fNIRS) neuroimaging in exercise-cognition science: a systematic, methodology-focused review. J. Clin. Med. 7, 12 (2018).
Xin Hu, Chu Zhuang, Fei Wang, Yong-Jin Liu, Chang-Hwan Im, and Dan Zhang. 2019. fNIRS evidence for recognizably different positive emotions. Front. Hum. Neurosci. 13 (2019).
Giulio Jacucci, Oswald Barral, Pedram Daee, Markus Wenzel, Baris Serim, Tuukka Ruotsalo, Patrik Pluchino, Jonathan Freeman, Luciano Gamberini, Samuel Kaski, et al. 2019. Integrating neurophysiologic relevance feedback in intent modeling for information retrieval. J. Assoc. Inf. Sci. Technol. 70, 9 (2019).
Thorsten Joachims. 2002. Optimizing search engines using clickthrough data. In Proc. KDD.
Lauri Kangassalo, Michiel Spapé, Giulio Jacucci, and Tuukka Ruotsalo. 2019. Why do users issue good queries? Neural correlates of term specificity. In Proc. SIGIR.
Lauri Kangassalo, Michiel Spapé, and Tuukka Ruotsalo. 2020. Neuroadaptive modelling for generating images matching perceptual categories. Sci. Rep. 10, 1 (2020).
Aditya Khosla, Atish Das Sarma, and Raffay Hamid. 2014. What makes an image popular?. In Proc. WWW.
Peter J. Lang, Margaret M. Bradley, and Bruce N. Cuthbert. 2008. International Affective Picture System (IAPS): Affective ratings of pictures and instruction manual.
Jana Machajdik and Allan Hanbury. 2010. Affective image classification using features inspired by psychology and art theory. In Proc. MM.
Daniel McDuff, Paul Thomas, Nick Craswell, Kael Rowan, and Mary Czerwinski. 2021. Do Affective Cues Validate Behavioural Metrics for Search?. In Proc. SIGIR.
Dominika Michalkova, Mario Parra-Rodriguez, and Yashar Moshfeghi. 2022. Information Need Awareness: an EEG study. In Proc. SIGIR.
Yashar Moshfeghi and Joemon M Jose. 2013. An effective implicit relevance feedback technique using affective, physiological and behavioural features. In Proc. SIGIR.
Sakrapee Paisalnan, Yashar Moshfeghi, and Frank Pollick. 2021. Neural Correlates of Realisation of Satisfaction in a Successful Search Process. Proc. Assoc. Inf. Sci. 58, 1 (2021).
Zuzana Pinkosova, William J. McGeown, and Yashar Moshfeghi. 2020. The Cortical Activity of Graded Relevance. In Proc. SIGIR.
Luca Pollonini, Cristen Olds, Homer Abaya, Heather Bortfeld, Michael S Beauchamp, and John S Oghalai. 2014. Auditory cortex activation to natural speech and simulated cochlear implant speech measured with functional near-infrared spectroscopy. Hear. Res. 309 (2014).
Tianrong Rao, Xiaoxu Li, and Min Xu. 2020. Learning multi-level deep representations for image emotion classification. Neural Process. Lett. 51, 3 (2020).
James A Russell. 1980. A circumplex model of affect. J. Pers. Soc. Psychol. 39, 6 (1980).
Stefanie Schmidt and Wolfgang G Stock. 2009. Collective indexing of emotions in images. A study in emotional information retrieval. J. Assoc. Inf. Sci. Technol. 60, 5 (2009).
Lucas R Trambaiolli, Juliana Tossato, André M Cravo, Claudinei E Biazoli Jr, and João R Sato. 2021. Subject-independent decoding of affective states using functional near-infrared spectroscopy. Plos one 16, 1 (2021), e0244840.
Anirudh Unni, Klas Ihme, Henrik Surm, Lars Weber, Andreas Lüdtke, Daniela Nicklas, Meike Jipp, and Jochem W Rieger. 2015. Brain activity measured with fNIRS for the prediction of cognitive workload. In Proc. CogInfoCom. IEEE.
Carmen Vidaurre and Benjamin Blankertz. 2010. Towards a Cure for BCI Illiteracy. Brain Topogr. 23, 2 (2010).
Ji Wan, Dayong Wang, Steven Chu Hong Hoi, Pengcheng Wu, Jianke Zhu, Yong-dong Zhang, and Jintao Li. 2014. Deep learning for content-based image retrieval: A comprehensive study. In Proc. MM.
Jiang Wang, Yang Song, Thomas Leung, Chuck Rosenberg, Jingbin Wang, James Philbin, Bo Chen, and Ying Wu. 2014. Learning fine-grained image similarity with deep ranking. In Proc. CVPR.
Victoria Yanulevskaya, Jan C van Gemert, Katharina Roth, Ann-Katrin Herbold, Nicu Sebe, and Jan-Mark Geusebroek. 2008. Emotional valence categorization using holistic image features. In Proc. ICIP.
Sicheng Zhao, Guiguang Ding, Qingming Huang, Tat-Seng Chua, Björn W Schuller, and Kurt Keutzer. 2018. Affective Image Content Analysis: A Comprehensive Survey. In Proc. IJCAI.
Sicheng Zhao, Yue Gao, Xiaolei Jiang, Hongxun Yao, Tat-Seng Chua, and Xiaoshuai Sun. 2014. Exploring principles-of-art features for image emotion recognition. In Proc. MM.
Sicheng Zhao, Xingxu Yao, Jufeng Yang, Guoli Jia, Guiguang Ding, Tat-Seng Chua, Bjoern W Schuller, and Kurt Keutzer. 2021. Affective image content analysis: Two decades review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intellig. (2021).
