Affective Relevance

Clickthrough data; Computational modelling; Conceptual understanding; Dwell time; Estimation methods; Information relevances; Model informations; Relevance estimations; Search engine results; Social media searches; Computer Networks and Communications; Artificial Intelligence; Physiology; Biomedical monitoring; Task analysis; Sensors; Monitoring; Intelligent systems; Modeling; Behavioral sciences; Decision making; Information analysis; User experience

Abstract :

[en] Modeling information relevance aims to construct a conceptual understanding of information significant for users' goals. Today, myriad relevance estimation methods are extensively used in various systems and services, mostly using behavioral signals such as dwell-time and click-through data and computational models of visual or textual correspondence to these behavioral signals. Consequently, these signals have become integral for personalizing social media, search engine results, and even supporting critical decision making. However, behavioral signals can only be used to produce rough estimations of the actual underlying affective states that users experience. Here, we provide an overview of recent alternative approaches for measuring and modeling more nuanced relevance based on physiological and neurophysiological sensing. Physiological and neurophysiological signals can directly measure users' affective responses to information and provide rich data that are not accessible via behavioral measurements. With these data, it is possible to account for users' affective experience and attentional correlates toward information.

Disciplines :

Computer science

Author, co-author :

Ruotsalo, Tuukka ; LUT University, Lahti, Finland ; University of Copenhagen, Copenhagen, Denmark

Traver, V. Javier ; INIT, Jaume I University, Castelló de la Plana, Spain

Kawala-Sterniuk, Aleksandra ; Opole University of Technology, Opole, Poland

LEIVA, Luis A. ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)

External co-authors :

yes

Language :

English

Title :

Affective Relevance

Publication date :

2024

Journal title :

IEEE Intelligent Systems

ISSN :

1541-1672

Publisher :

Institute of Electrical and Electronics Engineers Inc.

Volume :

Issue :

Pages :

12 - 22

Peer reviewed :

Peer Reviewed verified by ORBi

Additional URL :

http://xplorestaging.ieee.org/ielx7/9670/10614835/10506106.pdf?arnumber=10506106

European Projects :

HE - 101071147 - SYMBIOTIK - Context-aware adaptive visualizations for critical decision making

FnR Project :

FNR15722813 - BANANA - Brainsourcing For Affective Attention Estimation, 2021 (01/02/2022-31/01/2025) - Luis Leiva

Funders :

European Union

Funding text :

This work was supported by the Horizon 2020 FET program of the European Union (grant CHIST-ERA-20-BCI001) and the European Innovation Council Pathfinder program (SYMBIOTIK project, grant 101071147). This work was also supported by the Academy of Finland (grants 352915, 350323, 336085, 322653). This work is part of the project PCI2021- 122036-2A, funded by MCIN/AEI/10.13039/501100011033 and by the European Union NextGenerationEU/PRTR.

Available on ORBilu :

since 08 January 2026

Statistics

Number of views

19 (1 by Unilu)

Number of downloads

1 (0 by Unilu)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

OpenAlex citations

WoS citations^™

Bibliography

O. Barral et al., "Exploring peripheral physiology as a predictor of perceived relevance in information retrieval," in Proc. 20th Int. Conf. Intell. User Interfaces (IUI), New York, NY, USA, Association for Computing Machinery, 2015, pp. 389-399, doi: 10.1145/2678025. 2701389.
K. M. Davis, III, L. Kangassalo, M. Spape, and T. Ruotsalo, "Brainsourcing: Crowdsourcing recognition tasks via collaborative brain-computer interfacing," in Proc. ACM Conf. Human Factors Comput. Syst. (CHI), 2020, pp. 1-14, doi: 10.1145/ 3313831.3376288.
M. Frank et al., "Using EEG-based BCI devices to subliminally probe for private information," in Proc. Workshop Privacy Electron. Soc. (WPES), 2017, pp. 133-136, doi: 10.1145/3139550.3139559.
S. C. Leong, Y. M. Tang, C. H. Lai, and C. K. M. Lee, "Facial expression and body gesture emotion recognition: A systematic review on the use of visual data in affective computing," Comput. Sci. Rev., vol. 48, May 2023, Art. no. 100545, doi: 10.1016/ j.cosrev.2023.100545.
J. Z. Lim, J. Mountstephens, and J. Teo, "Emotion recognition using eye-tracking: Taxonomy, review and current challenges," Sensors, vol. 20, no. 8, 2020, Art. no. 2384, doi: 10.3390/s20082384.
R. Martinek et al., "Advanced bioelectrical signal processing methods: Past, present, and future approach-Part III: Other biosignals," Sensors, vol. 21, no. 18, 2021, Art. no. 6064, doi: 10.3390/s21186064.
D. McDuff, R. e. Kaliouby, T. Senechal, M. Amr, J. F. Cohn, and R. Picard, "Affectiva-MIT facial expression dataset (AM-FED): Naturalistic and spontaneous facial expressions collected 'in-the-wild,'" in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) Workshops, Jun. 2013, pp. 881-888.
J. Mizgajski and M. Morzy, "Affective recommender systems in online news industry: How emotions influence reading choices," User Model. User-Adapted Interact, vol. 29, no. 2, pp. 345-379, 2019, doi: 10.1007/ s11257-018-9213-x.
M. Nardelli, A. Greco, L. Sebastiani, and E. P. Scilingo, "ComEDA: A new tool for stress assessment based on electrodermal activity," Comput. Biol. Med., vol. 150, Nov. 2022, Art. no. 106144, doi: 10.1016/ j.compbiomed.2022.106144.
R. Plutchik, "A general psychoevolutionary theory of emotion," in Theories of Emotion. New York, NY, USA: Elsevier, 1980, pp. 3-33.
S. Prange, S. Mayer, M.-L. Bittl, M. Hassib, and F. Alt, "Investigating user perceptions towards wearable mobile electromyography," in Proc. IFIP Conf. Human-Comput. Interaction (INTERACT), 2021, pp. 339-360.
Z. Ren et al., "VEATIC: Video-based emotion and affect tracking in context dataset," in Proc. IEEE/CVF Winter Conf. Appl. Comput. Vis. (WACV), Jan. 2024, pp. 4467-4477.
J. Sabio, N. S. Williams, G. M. McArthur, and N. A. Badcock, "A scoping review on the use of consumergrade EEG devices for research," PLoS One, vol. 19, no. 3, 2024, Art. no. e0291186, doi: 10.1371/journal. pone.0291186.
N. Selvaraj, A. Jaryal, J. Santhosh, K. K. Deepak, and S. Anand, "Assessment of heart rate variability derived from finger-tip photoplethysmography as compared to electrocardiography," J. Med. Eng. Technol., vol. 32, no. 6, 2008, doi: 10.1080/ 03091900701781317.
Y. B. Singh and S. Goel, "A systematic literature review of speech emotion recognition approaches," Neurocomputing, vol. 492, pp. 245-263, Jul. 2022, doi: 10.1016/j.neucom.2022.04.028.
Ruotsalo, T. Makela, K. S. M. Miciel, and L. A. Luis, "Feeling positive? Predicting emotional image similarity from brain signals," in Proc. 31st ACM Int. Conf. Multimedia, 2023, pp. 5870-5878, doi: 10.1145/ 3581783.3613442.
S. Wang, L. Zhong, Y. Fu, L. Chen, J. Ren, and Y. Zhang, "Uface: Your smartphone can "hear" your facial expression!" in Proc. ACM Interactive Mobile Wearable Ubiquitous Technol., Mar. 2024, vol. 8, no. 1, pp. 1-27, doi: 10.1145/3643546.
W. Yang and Y. Xie, "Can robots elicit empathy? The effects of social robots' appearance on emotional contagion," Comput. Human Behav., Artif. Humans, vol. 2, no. 1, 2024, Art. no. 100049, doi: 10.1016/j.chbah. 2024.100049.
J. Zhang, Z. Yin, P. Chen, and S. Nichele, "Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review," Inf. Fusion, vol. 59, pp. 103-126, Jul. 2020, doi: 10.1016/ j.inffus.2020.01.011.
S. Zhao et al., "Affective image content analysis: Two decades review and new perspectives," IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 10, pp. 6729-6751, Oct. 2021.