Conversational Agents Trust Calibration: A User-Centred Perspective to Design

DUBIEL, Mateusz; Daronnat, Sylvain; LEIVA, Luis A.

doi:10.1145/3543829.3544518

Download

Paper published in a book (Scientific congresses, symposiums and conference proceedings)

Conversational Agents Trust Calibration: A User-Centred Perspective to Design

DUBIEL, Mateusz; Daronnat, Sylvain; LEIVA, Luis A.

2022 • In ACM Conference on Conversational User Interfaces (CUI 2022)

Peer reviewed

Permalink
https://hdl.handle.net/10993/51739

DOI
10.1145/3543829.3544518

Files (1)Send to Details Statistics Bibliography Similar publications

Files

Full Text

CUI22_Trust_Calibration.pdf

Author postprint (456.59 kB)

Download

All documents in ORBilu are protected by a user license.

Send to

RIS BibTex APA Chicago Permalink X Linkedin

Details

Keywords :

Conversational Agents; Trust; Design Ethics

Abstract :

[en] Previous work identified trust as one of the key requirements for adoption and continued use of conversational agents (CAs). Given recent advances in natural language processing and deep learning, it is currently possible to execute simple goal-oriented tasks by using voice. As CAs start to provide a gateway for purchasing products and booking services online, the question of trust and its impact on users’ reliance and agency becomes ever-more pertinent. This paper collates trust-related literature and proposes four design suggestions that are illustrated through example conversations. Our goal is to encourage discussion on ethical design practices to develop CAs that are capable of employing trust-calibration techniques that should, when relevant, reduce the user’s trust in the agent. We hope that our reflections, based on the synthesis of insights from the fields of human-agent interaction, explainable ai, and information retrieval, can serve as a reminder of the dangers of excessive trust in automation and contribute to more user-centred CA design.

Disciplines :

Computer science

Author, co-author :

DUBIEL, Mateusz ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)

Daronnat, Sylvain; University of Strathclyde

LEIVA, Luis A. ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)

External co-authors :

yes

Language :

English

Title :

Conversational Agents Trust Calibration: A User-Centred Perspective to Design

Publication date :

2022

Event name :

4th Conference on Conversational User Interfaces (CUI 2022)

Event organizer :

Association for Computing Machinery (ACM)

Event place :

Glasgow, United Kingdom

Event date :

from 26-07-2022 to 25-07-2022

Audience :

International

Main work title :

ACM Conference on Conversational User Interfaces (CUI 2022)

Publisher :

ACM

Peer reviewed :

Peer reviewed

Focus Area :

Computational Sciences

FnR Project :

FNR15722813 - Brainsourcing For Affective Attention Estimation, 2021 (01/02/2022-31/01/2025) - Luis Leiva

Available on ORBilu :

since 20 July 2022

Statistics

Number of views

234 (42 by Unilu)

Number of downloads

215 (7 by Unilu)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

OpenAlex citations

Bibliography

Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977 (2020).
Kamran Alipour, Jurgen P Schulze, Yi Yao, Avi Ziskind, and Giedrius Burachas. 2020. A study on multimodal and interactive explanations for visual question answering. arXiv preprint arXiv:2003.00431 (2020).
Jorge A. Alvarado-Valencia and Lope H. Barrero. 2014. Reliance, trust and heuristics in judgmental forecasting. Computers in Human Behavior 36 (2014), 102-113. https://doi.org/10.1016/j.chb.2014.03.047
Tawfiq Ammari, Jofish Kaye, Janice Y Tsai, and Frank Bentley. 2019. Music, Search, and IoT: How People (Really) Use Voice Assistants. ACM Trans. Comput. Hum. Interact. 26, 3 (2019), 17-1.
Salvatore Andolina, Valeria Orso, Hendrik Schneider, Khalil Klouche, Tuukka Ruotsalo, Luciano Gamberini, and Giulio Jacucci. 2018. Investigating Proactive Search Support in Conversations. In Proceedings of the 2018 Designing Interactive Systems Conference. ACM, 1295-1307.
Matthew P Aylett, Selina Jeanne Sutton, and Yolanda Vazquez-Alvarez. 2019. The right kind of unnatural: designing a robot voice. In Proceedings of the 1st International Conference on Conversational User Interfaces. 1-2.
Marcia J Bates. 1990. Where should the person stop and the information search interface start' Information Processing & Management 26, 5 (1990), 575-591.
Rodrigo Bavaresco, Diórgenes Silveira, Eduardo Reis, Jorge Barbosa, Rodrigo Righi, Cristiano Costa, Rodolfo Antunes, Marcio Gomes, Clauter Gatti, Mariangela Vanzin, Saint Clair Junior, Elton Silva, and Carlos Moreira. 2020. Conversational agents in business: A systematic literature review and future research directions. Computer Science Review 36 (2020), 100239. https://doi.org/10.1016/j.cosrev.2020. 100239
Jessie Y Chen, Katelyn Procci, Michael Boyce, Julia Wright, Andre Garcia, and Michael Barnes. 2014. Situation awareness-based agent transparency. Technical Report. Army research lab aberdeen proving ground md human research and engineering . . . .
Aleksandr Chuklin, Aliaksei Severyn, Johanne Trippas, Enrique Alfonseca, Hanna Silen, and Damiano Spina. 2018. Prosody modifications for question-Answering in voice-only settings. arXiv preprint arXiv:1806.03957 (2018).
Leigh Clark, Nadia Pantidi, Orla Cooney, Philip Doyle, Diego Garaialde, Justin Edwards, Brendan Spillane, Emer Gilmartin, Christine Murad, Cosmin Munteanu, et al. 2019. What makes a good conversation' Challenges in designing truly conversational agents. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1-12.
Benjamin R Cowan, Nadia Pantidi, David Coyle, Kellie Morrissey, Peter Clarke, Sara Al-Shehri, David Earley, and Natasha Bandeira. 2017. " What can i help you with'" infrequent users' experiences of intelligent personal assistants. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services. 1-12.
Ewart J De Visser, Samuel S Monfort, Ryan McKendrick, Melissa AB Smith, Patrick E McKnight, Frank Krueger, and Raja Parasuraman. 2016. Almost human: Anthropomorphism increases trust resilience in cognitive agents. Journal of Experimental Psychology: Applied 22, 3 (2016), 331.
Berkeley J Dietvorst, Joseph P Simmons, and Cade Massey. 2015. Algorithm aversion: people erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General 144, 1 (2015), 114.
Mateusz Dubiel, Martin Halvey, Leif Azzopardi, Damien Anderson, and Sylvain Daronnat. 2020. Conversational strategies: impact on search performance in a goal-oriented task. In The Third International Workshop on Conversational Approaches to Information Retrieval.
Mateusz Dubiel, Martin Halvey, Pilar Oplustil Gallegos, and Simon King. 2020. Persuasive synthetic speech: Voice perception and user behaviour. In Proceedings of the 2nd Conference on Conversational User Interfaces. 1-9.
Justin Edwards and Elaheh Sanoubari. 2019. A need for trust in conversational interface research. In Proceedings of the 1st International Conference on Conversational User Interfaces. 1-3.
Aaron C Elkins and Douglas C Derrick. 2013. The sound of trust: voice as a measurement of trust during interactions with embodied conversational agents. Group decision and negotiation 22, 5 (2013), 897-913.
Xiaocong Fan, Sooyoung Oh, Michael McNeese, John Yen, Haydee Cuevas, Laura Strater, and Mica R. Endsley. 2008. The influence of agent reliability on trust in human-Agent collaboration. Proceedings of the 15th European conference on Cognitive ergonomics the ergonomics of cool interaction-ECCE '08 (2008), 1. https: //doi.org/10.1145/1473018.1473028
Andrew Gibiansky, Sercan Omer Arik, Gregory Frederick Diamos, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou. 2017. Deep Voice 2: Multi-Speaker Neural Text-To-Speech. In NIPS.
Yunha Han. 2021. Wait, Let's Think about Your Purchase Again: A Study on Interventions for Supporting Self-Controlled Online Purchases. (2021). To Appear in Proceedings of Web Conference 2021.
Kevin Anthony Hoff and Masooda Bashir. 2015. Trust in automation: Integrating empirical evidence on factors that influence trust. Human Factors 57, 3 (2015), 407-434. https://doi.org/10.1177/0018720814547570
Philipp Kirschthaler, Martin Porcheron, and Joel E Fischer. 2020. What can i say' effects of discoverability in vuis on task performance and user experience. In Proceedings of the 2nd Conference on Conversational User Interfaces. 1-9.
Alexander Kunze, Stephen J. Summerskill, Russell Marshall, and Ashleigh J. Filtness. 2019. Automation transparency: implications of uncertainty communication for human-Automation interaction and interfaces. Ergonomics 62, 3 (2019), 345-360. https://doi.org/10.1080/00140139.2018.1547842
J. D. Lee and K. A. See. 2004. Trust in Automation: Designing for Appropriate Reliance. Human Factors: The Journal of the Human Factors and Ergonomics Society 46, 1 (2004), 50-80. https://doi.org/10.1518/hfes.46.1.50-30392
Rosemarijn Looije, Mark A Neerincx, and Fokie Cnossen. 2010. Persuasive robotic assistant for health self-management of older adults: Design and evaluation of social behaviors. International Journal of Human-Computer Studies 68, 6 (2010), 386-397.
Ewa Luger and Abigail Sellen. 2016. " Like Having a Really Bad PA" The Gulf between User Expectation and Experience of Conversational Agents. In Proceedings of the 2016 CHI conference on human factors in computing systems. 5286-5297.
D. Manzey, J Elin Bahner, and Anke-Dorothea Hueper. 2006. Misuse of Automated Aids in Process Control: Complacency, Automation Bias and Possible Training Interventions. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 50, 3 (2006), 220-224. https://doi.org/10.1177/154193120605000303
John M. McGuirl and Nadine B. Sarter. 2006. Supporting Trust Calibration and the Effective Use of Decision Aids by Presenting Dynamic System Confidence Information. Human Factors 48, 4 (2006), 656-665. https://doi.org/10.1518/001872006779166334 arXiv:https://doi.org/10.1518/001872006779166334 PMID: 17240714.
Bonnie M. Muir. 1987. Trust between humans and machines, and the design of decision aids. International Journal of Man-Machine Studies 27, 5-6 (1987), 527-539. https://doi.org/10.1016/S0020-7373(87)80013-5
M. Ng, K. P. L. Coopamootoo, E. Toreini, M. Aitken, K. Elliot, and A. van Moorsel. 2020. Simulating the Effects of Social Presence on Trust, Privacy Concerns Usage Intentions in Automated Bots for Finance. In 2020 IEEE European Symposium on Security and Privacy Workshops (EuroS PW). 190-199. https://doi.org/10.1109/EuroSPW51379.2020.00034
John O'Donovan and Barry Smyth. 2005. Trust in recommender systems. In Proceedings of the 10th international conference on Intelligent user interfaces. 167-174.
Thomas C O'Guinn and Ronald J Faber. 1989. Compulsive buying: A phenomenological exploration. Journal of consumer research 16, 2 (1989), 147-157.
Richard Pak, Nicole Fink, Margaux Price, Brock Bass, and Lindsay Sturre. 2012. Decision support aids with anthropomorphic characteristics influence trust and performance in younger and older adults. Ergonomics 55, 9 (2012), 1059-1072.
Raja Parasuraman, Thomas B Sheridan, and Christopher D Wickens. 2000. A model for types and levels of human interaction with automation. IEEE Transactions on systems, man, and cybernetics-Part A: Systems and Humans 30, 3 (2000), 286-297.
Claudio S Pinhanez. 2021. Expose Uncertainty, Instill Distrust, Avoid Explanations: Towards Ethical Guidelines for AI. arXiv preprint arXiv:2112.01281 (2021).
Lingyun Qiu and Izak Benbasat. 2009. Evaluating anthropomorphic product recommendation agents: A social relationship perspective to designing information systems. Journal of management information systems 25, 4 (2009), 145-182.
Minjin Rheu, Ji Youn Shin, Wei Peng, and Jina Huh-Yoo. 2021. Systematic review: Trust-building factors and implications for conversational agent design. International Journal of Human-Computer Interaction 37, 1 (2021), 81-96.
Cynthia Rudin. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence 1, 5 (2019), 206-215.
Dominik Sacha, Hansi Senaratne, Bum Chul Kwon, Geoffrey Ellis, and Daniel A. Keim. 2016. The Role of Uncertainty, Awareness, and Trust in Visual Analytics. IEEE Transactions on Visualization and Computer Graphics 22, 1 (2016), 240-249. https://doi.org/10.1109/TVCG.2015.2467591
Pararth Shah, Dilek Hakkani-Tur, Bing Liu, and Gokhan Tur. 2018. Bootstrapping a neural conversational agent with dialogue self-play, crowdsourcing and on-line reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers). 41-51.
Ameneh Shamekhi, Q Vera Liao, Dakuo Wang, Rachel KE Bellamy, and Thomas Erickson. 2018. Face Value' Exploring the effects of embodiment for a group facilitation agent. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1-13.
Ben Shneiderman. 2020. Human-centered artificial intelligence: Reliable, safe & trustworthy. International Journal of Human-Computer Interaction 36, 6 (2020), 495-504.
Kimberly Stowers, Nicholas Kasdaglis, Michael Rupp, Jessie Chen, Daniel Barber, and Michael Barnes. 2017. Insights into Human-Agent Teaming: Intelligent Agent Transparency and Uncertainty. In Advances in Human Factors in Robots and Unmanned Systems, Pamela Savage-Knepshield and Jessie Chen (Eds.). Springer International Publishing, Cham, 149-160.
Eva Szekely, Gustav Eje Henter, Jonas Beskow, and Joakim Gustafson. 2019. Spontaneous Conversational Speech Synthesis from Found Data. In INTERSPEECH. 4435-4439.
Madiha Tabassum, Tomasz Kosinski, Alisa Frik, Nathan Malkin, Primal Wijesekera, Serge Egelman, and Heather Richter Lipford. 2019. Investigating Users' Preferences and Expectations for Always-Listening Voice Assistants. Proceedings of the ACM on Interactive, Mobile,Wearable and Ubiquitous Technologies 3, 4 (2019), 1-23.
Ilaria Torre, Jeremy Goslin, Laurence White, and Debora Zanatto. 2018. Trust in artificial voices: A" congruency effect" of first impressions and behavioural experience. In Proceedings of the Technology, Mind, and Society. 1-6.
Johanne R Trippas. 2019. Spoken Conversational Search: Audio-only Interactive Information Retrieval. Ph.D. Dissertation. PhD thesis, RMIT, Melbourne.
Nigel Ward, Jonathan E Avila, and Aaron M Alarcon. 2021. Towards Continuous Estimation of Dissatisfaction in Spoken Dialog. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue. 13-20.
Nigel G Ward. 2019. Prosodic patterns in English conversation. Cambridge University Press.
Nigel G Ward, Ambika Kirkland, Marcin Wlodarczak, and Eva Szekely. [n.d.]. Two Pragmatic Functions of Breathy Voice in American English Conversation. ([n. d.]).
Katharina Weitz, Dominik Schiller, Ruben Schlagowski, Tobias Huber, and Elisabeth Andre. 2020. "Let me explain!": exploring the potential of virtual agents in explainable AI interaction design. Journal on Multimodal User Interfaces (2020), 1-12.
Ryen W White and Ian Ruthven. 2006. A study of interface support mechanisms for interactive information retrieval. Journal of the American Society for Information Science and Technology 57, 7 (2006), 933-948.
Christine T. Wolf and Kathryn E. Ringland. 2020. Designing Accessible, Explainable AI (XAI) Experiences. SIGACCESS Access. Comput. 125, Article 6 (March 2020), 1 pages. https://doi.org/10.1145/3386296.3386302
Wayne Xiong, Jasha Droppo, Xuedong Huang, Frank Seide, Mike Seltzer, Andreas Stolcke, Dong Yu, and Geoffrey Zweig. 2016. Achieving human parity in conversational speech recognition. arXiv preprint arXiv:1610.05256 (2016).
Sunghwan Yi and Hans Baumgartner. 2011. Coping with guilt and shame in the impulse buying context. Journal of Economic Psychology 32, 3 (2011), 458-467.
Debora Zanatto, Massimiliano Patacchiola, Jeremy Goslin, and Angelo Cangelosi. 2016. Priming anthropomorphism: Can the credibility of humanlike robots be transferred to non-humanlike robots'. In 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, 543-544.