Feeling Positive? Predicting Emotional Image Similarity from Brain Signals

bci; ranking relevance; Affective appraisal; Affective Computing; Brain signals; Functional near infrared spectroscopy; Image content; Image similarity; Visual content

Abstract :

[en] The present notion of visual similarity is based on features derived from image contents. This ignores the users' emotional or affective experiences toward the content, and how users feel when they search for images. Here we consider valence, a positive or negative quantification of affective appraisal, as a novel dimension of image similarity. We report the largest neuroimaging experiment that quantifies and predicts the valence of visual content by using functional near-infrared spectroscopy from brain-computer interfacing. We show that affective similarity can be (1)∼decoded directly from brain signals in response to visual stimuli, (2)∼utilized for predicting affective image similarity with an average accuracy of 0.58 and an accuracy of 0.65 for high-arousal stimuli, and (3)∼effectively used to complement affective similarity estimates of content-based models; for example when fused fNIRS and image rankings the retrieval F-measure@20 is 0.70. Our work opens new research avenues for affective multimedia analysis, retrieval, and user modeling.

Disciplines :

Computer science

Author, co-author :

Ruotsalo, Tuukka ; University of Copenhagen, LUT University, Finland

Mäkelä, Kalle ; University of Helsinki, Helsinki, Finland

Spapé, Michiel M. ; University of Helsinki, Helsinki, Finland

LEIVA, Luis A. ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)

External co-authors :

yes

Language :

English

Title :

Feeling Positive? Predicting Emotional Image Similarity from Brain Signals

Publication date :

26 October 2023

Event name :

Proceedings of the 31st ACM International Conference on Multimedia

Event place :

Ottawa, Can

Event date :

29-10-2023 => 03-11-2023

Audience :

International

Main work title :

MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia

Publisher :

Association for Computing Machinery, Inc

ISBN/EAN :

9798400701085

Peer reviewed :

Peer reviewed

Additional URL :

https://dl.acm.org/doi/pdf/10.1145/3581783.3613442

European Projects :

HE - 101071147 - SYMBIOTIK - Context-aware adaptive visualizations for critical decision making

FnR Project :

FNR15722813 - Brainsourcing For Affective Attention Estimation, 2021 (01/02/2022-31/01/2025) - Luis Leiva

Funders :

Union Européenne

Funding text :

This work is supported by the Academy of Finland (grants 352915, 350323, 336085, 322653), the Horizon 2020 FET program of the European Union through the ERA-NET Cofund funding grant CHISTERA-20-BCI-001, and the European Innovation Council Pathfinder program (SYMBIOTIK project, grant 101071147).

Available on ORBilu :

since 20 March 2024

Statistics

Number of views

72 (3 by Unilu)

Number of downloads

56 (1 by Unilu)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenAlex citations

Bibliography

A. R. Andreu-Perez, M. Kiani, J. Andreu-Perez, P. Reddy, J. Andreu-Abela, M. Pinto, and K. Izzetoglu. 2021. Single-Trial Recognition of Video Gamer's Expertise from Brain Haemodynamic and Facial Emotion Responses. Brain Sci. 11, 1 (2021).
I. Arapakis, J. M. Jose, and P. D. Gray. 2008. Affective Feedback: An Investigation into the Role of Emotions in the Information Seeking Process. In Proc. SIGIR.
P. Avero and M. G. Calvo. 2006. Affective priming with pictures of emotional scenes: The role of perceptual similarity and category relatedness. Span. J. Psychol. 9, 1 (2006).
H. Ayaz, M. Izzetoglu, K. Izzetoglu, and B. Onaral. 2019. The use of functional near-infrared spectroscopy in neuroergonomics. In Neuroergonomics.
M. Balconi, E. Grippa, and M. E. Vanutelli. 2015. What hemodynamic (fNIRS), electrophysiological (EEG) and autonomic integrated measures can tell us about emotional processing. Brain Cogn. 95 (2015).
D. Bandara, L. Hirshfield, and S. Velipasalar. 2019. Classification of affect using deep learning on brain blood flow data. J. Near Infrared Spectrosc. 27, 3 (2019).
S. C. Bunce, M. Izzetoglu, K. Izzetoglu, B. Onaral, and K. Pourrezaei. 2006. Functional near-infrared spectroscopy. IEEE Eng. Med. Biol. Mag. 25, 4 (2006).
L. Carretié. 2014. Exogenous (automatic) attention to emotional stimuli: A review. Cogn. Affect. Behav. Neurosci. 14, 4 (2014).
M. Chen, L. Zhang, and J. P. Allebach. 2015. Learning deep features for image emotion classification. In Proc. ICIP.
Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu. 2020. Uniter: Universal image-text representation learning. In Proc. ECCV.
G. V. Cormack, C. L. A. Clarke, and S. Buettcher. 2009. Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. In Proc. SIGIR.
K. M. Davis, M. Spapé, and T. Ruotsalo. 2022. Contradicted by the Brain: Predicting Individual and Group Preferences via Brain-Computer Interfacing. IEEE Trans. Affect. Comput. (2022).
K. M. Davis III, M. Spapé, and T. Ruotsalo. 2021. Collaborative Filtering with Preferences Inferred from Brain Signals. In Proc. WWW.
C. de la Torre-Ortiz, M. Spapé, and T. Ruotsalo. 2023. The P3 indexes the distance between perceived and target image. Psychophysiology 60, 5 (2023), e14225.
D. T. Delpy, M. Cope, P. van der Zee, S. Arridge, S. Wray, and J. Wyatt. 1988. Estimation of optical pathlength through tissue from direct time of flight measurement. Phys. Med. Biol. 33, 12 (1988).
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proc. NAACL.
F. A. Fishburn, R. S. Ludlum, C. J. Vaidya, and A. V. Medvedev. 2019. Temporal derivative distribution repair (TDDR): A motion correction method for fNIRS. Neuroimage 184 (2019).
K. He, X. Zhang, S. Ren, and J. Sun. 2016. Deep Residual Learning for Image Recognition. In Proc. CVPR.
D. Heger, C. Herff, F. Putze, R. Mutter, and T. Schultz. 2014. Continuous affective states recognition using functional near infrared spectroscopy. Brain Comput. Interfaces 1, 2 (2014).
X. Hu, C. Zhuang, F. Wang, Y.-J. Liu, C.-H. Im, and D. Zhang. 2019. fNIRS evidence for recognizably different positive emotions. Front. Hum. Nneurosci. 13 (2019).
R. Jenke, A. Peer, and M. Buss. 2014. Feature extraction and selection for emotion recognition from EEG. IEEE Trans. Affect. Comput. 5, 3 (2014).
K. S. Jones, S. Walker, and S. E. Robertson. 2000. A probabilistic model of information retrieval: Development and comparative experiments. Inf. Proces. Manag. 36, 6 (2000).
G. Kazai, P. Thomas, and N. Craswell. 2019. The Emotion Profile of Web Search. In Proc. SIGIR.
J. Kenemans, M. Verbaten, W. Sjouw, and J. Slangen. 1988. Effects of task relevance on habituation of visual single-trial ERPs and the skin conductance orienting response. Int. J. Psychophysiol. 6, 1 (1988).
A. Khosla, A. Das Sarma, and R. Hamid. 2014. What makes an image popular?. In Proc. WWW.
S. Koelstra, C. Muhl, M. Soleymani, J.-S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Patras. 2011. DEAP: A database for emotion analysis using physiological signals. IEEE Trans. Affect. Comput. 3, 1 (2011).
P. J. Lang, M. M. Bradley, and B. N. Cuthbert. 2008. International affective picture system (IAPS): Technical manual and affective ratings. Technical Report A-8. NIMH Center for the Study of Emotion and Attention.
Q. Le and T. Mikolov. 2014. Distributed representations of sentences and documents. In Proc. ICML.
L. A. Leiva, S. Morteza, and A. Oulasvirta. 2022. Modeling How Different User Groups Perceive Webpage Aesthetics. Univers. Access Inf. Soc. (2022).
T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Dollár. 2014. Microsoft COCO: Common Objects in Context. In Proc. ECCV.
L. Liu and Z. Chen. 2021. The Application of Closed-loop Brain Training: Nearinfrared Spectroscopy (NIRS) Neurofeedback. In Proc. ISAIMS.
Z. Liu, J. Shore, M. Wang, F. Yuan, A. Buss, and X. Zhao. 2021. A systematic review on hybrid EEG/fNIRS in brain-computer interface. Biomed. Signal Process. Control 68 (2021).
J. Machajdik and A. Hanbury. 2010. Affective image classification using features inspired by psychology and art theory. In Proc. MM.
S. Patil and S. Talbar. 2010. Content Based Image Retrieval Using Various Distance Metrics. In Proc. ICDEM. LNCS 6411.
P. Pinti, I. Tachtsidis, A. Hamilton, J. Hirsch, C. Aichelburg, S. Gilbert, and P. W. Burgess. 2020. The present and future use of functional near-infrared spectroscopy (fNIRS) for cognitive neuroscience. Ann. N. Y. Acad. Sci. 1464, 1 (2020).
B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik. 2017. Flickr30K Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models. Int. J. Comput. Vis. 123, 1 (2017).
L. Pollonini, C. Olds, H. Abaya, H. Bortfeld, M. S. Beauchamp, and J. S. Oghalai. 2014. Auditory cortex activation to natural speech and simulated cochlear implant speech measured with functional near-infrared spectroscopy. Hear. Res. 309 (2014).
K. Qing, R. Huang, and K.-S. Hong. 2021. Decoding Three Different Preference Levels of Consumers Using Convolutional Neural Network: A Functional Near-Infrared Spectroscopy Study. Front. Hum. Neurosci. (2021).
T. Rao, X. Li, and M. Xu. 2020. Learning multi-level deep representations for image emotion classification. Neural Process. Lett. 51, 3 (2020).
C. Redies, M. Grebenkina, M. Mohseni, A. Kaduhm, and C. Dobel. 2020. Global image properties predict ratings of affective pictures. Front. Psychol. 11 (2020).
N. Reimers and I. Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proc. EMNLP-IJCNLP.
T. Ruotsalo, K. Mäkelä, and M. Spape. 2023. Crowdsourcing Affective Annotations via fNIRS-BCI. IEEE Trans. Affect. Comput. (2023).
T. Ruotsalo, K. Mäkelä, M. M. Spapé, and L. A. Leiva. 2023. Affective Relevance: Inferring Emotional Responses via fNIRS Neuroimaging. In Proc. SIGIR.
G. Schalk and E. C. Leuthardt. 2011. Brain-computer interfaces using electrocorticographic signals. IEEE Rev. Biomed. Eng. 4 (2011).
W. C. Shipley, J. I. Coffin, and K. C. Hadsell. 1945. Affective distance and other factors determining reaction time in judgments of color preference. J. Exp. Psychol. 35, 3 (1945).
K. Simonyan and A. Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In Proc. ICLR.
A. Smeulders, M. Worring, S. Santini, A. Gupta, and R. Jain. 2000. Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intellig. 22, 12 (2000).
M. Spape, K. Mäkelä, and T. Ruotsalo. 2023. NEMO: A Database for Emotion Analysis Using Functional Near-infrared Spectroscopy. Preprint. To appear. (2023).
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. 2015. Going Deeper with Convolutions. In Proc. CVPR.
K. Tai and T. Chau. 2009. Single-trial classification of NIRS signals during emotional induction tasks: Towards a corporeal machine interface. J. Neuroeng. Rehabil. 6, 1 (2009).
L. R. Trambaiolli, A. Tiwari, and T. H. Falk. 2021. Affective neurofeedback under naturalistic conditions: A mini-review of current achievements and open challenges. Front. Neuroergonomics 2 (2021).
L. R. Trambaiolli, J. Tossato, A. M. Cravo, C. E. Biazoli Jr, and J. R. Sato. 2021. Subject-independent decoding of affective states using functional near-infrared spectroscopy. PLOS One 16, 1 (2021).
C. Vidaurre and B. Blankertz. 2010. Towards a Cure for BCI Illiteracy. Brain Topogr. 23, 2 (2010).
J. Wan, D. Wang, S. C. H. Hoi, P. Wu, J. Zhu, Y. Zhang, and J. Li. 2014. Deep learning for content-based image retrieval: A comprehensive study. In Proc. MM.
J. Wang, Y. Song, T. Leung, C. Rosenberg, J. Wang, J. Philbin, B. Chen, and Y. Wu. 2014. Learning fine-grained image similarity with deep ranking. In Proc. CVPR.
W. Webber, A. Moffat, and J. Zobel. 2010. A Similarity Measure for Indefinite Rankings. ACM Trans. Inf. Syst. 28, 4 (2010).
V. Yanulevskaya, J. C. van Gemert, K. Roth, A.-K. Herbold, N. Sebe, and J.-M. Geusebroek. 2008. Emotional valence categorization using holistic image features. In Proc. ICIP.
B. A. Yilma and L. A. Leiva. 2023. The Elements of Visual Art Recommendation: Learning Latent Semantic Representations of Paintings. In Proc. CHI.
Q. You, J. Luo, H. Jin, and J. Yang. 2016. Building a large scale dataset for image emotion recognition: The fine print and the benchmark. In Proc. AAAI.
M. A. Yücel, A. v. Lühmann, F. Scholkmann, J. Gervain, I. Dan, H. Ayaz, D. Boas, R. J. Cooper, J. Culver, C. E. Elwell, A. Eggebrecht, M. A. Franceschini, C. Grova, F. Homae, F. Lesage, H. Obrig, I. Tachtsidis, S. Tak, Y. Tong, A. Torricelli, H. Wabnitz, and M. Wolfc. 2021. Best practices for fNIRS publications. Neurophotonics 8, 1 (2021).
S. Zhao, G. Ding, Q. Huang, T.-S. Chua, B. W. Schuller, and K. Keutzer. 2018. Affective Image Content Analysis: A Comprehensive Survey. In Proc. IJCAI.
S. Zhao, Y. Gao, X. Jiang, H. Yao, T.-S. Chua, and X. Sun. 2014. Exploring principlesof-art features for image emotion recognition. In Proc. MM.
S. Zhao, X. Yao, J. Yang, G. Jia, G. Ding, T.-S. Chua, B. W. Schuller, and K. Keutzer. 2021. Affective image content analysis: Two decades review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intellig. (2021).