The Jiminy Advisor: Moral Agreements among Stakeholders Based on Norms and Argumentation

Collaborative decision making; Context-sensitive; Ethical behavior; Formal argumentation; Normative system; Artificial Intelligence; Computer Science - Artificial Intelligence

Abstract :

[en] An autonomous system is constructed by a manufacturer, operates in a society subject to norms and laws, and interacts with end users. All of these actors are stakeholders affected by the behavior of the autonomous system. We address the challenge of how the ethical views of such stakeholders can be integrated in the behavior of an autonomous system. We propose an ethical recommendation component called Jiminy which uses techniques from normative systems and formal argumentation to reach moral agreements among stakeholders. A Jiminy represents the ethical views of each stakeholder by using normative systems, and has three ways of resolving moral dilemmas that involve the opinions of the stakeholders. First, the Jiminy considers how the arguments of the stakeholders relate to one another, which may already resolve the dilemma. Secondly, the Jiminy combines the normative systems of the stakeholders such that the combined expertise of the stakeholders may resolve the dilemma. Thirdly, and only if these two other methods have failed, the Jiminy uses context-sensitive rules to decide which of the stakeholders take preference over the others. At the abstract level, these three methods are characterized by adding arguments, adding attacks between arguments, and revising attacks between arguments. We show how a Jiminy can be used not only for ethical reasoning and collaborative decision-making, but also to provide explanations about ethical behavior.

Disciplines :

Computer science

Author, co-author :

Liao, Beishui ; Zheijang University, School of Philosophy, Hangzhou, China ; The State Key Lab of Brain-Machine Intelligence, Hangzhou, China

PARDO VENTURA, Pere ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)

Slavkovik, Marija ; University of Bergen, Deptartment of Information Science and Media Studies, Bergen, Norway

VAN DER TORRE, Leon ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)

External co-authors :

yes

Language :

English

Title :

The Jiminy Advisor: Moral Agreements among Stakeholders Based on Norms and Argumentation

Publication date :

2023

Journal title :

Journal of Artificial Intelligence Research

ISSN :

1076-9757

eISSN :

1943-5037

Publisher :

AI Access Foundation

Volume :

Pages :

737 - 792

Peer reviewed :

Peer Reviewed verified by ORBi

Focus Area :

Computational Sciences

Additional URL :

https://www.jair.org/index.php/jair/article/download/14368/26944

FnR Project :

FNR14776480 - Deontic Logic For Epistemic Rights, 2020 (01/09/2021-31/08/2024) - Leon Van Der Torre

Name of the research project :

Deontic Logic for Epistemic Rights

Funding number :

OPEN O20/14776480,

Funding text :

A shorter version of this paper was presented at the AAAI/ACM Conference on Artificial Intelligence, Ethics and Society, January 27–28, 2019. That paper presents the general idea of a Jiminy advisor, but not the details of the formal argumentation mechanism. The authors are grateful to Dr. Louise Dennis for providing some of the examples in this paper. Beishui Liao was partially supported by the ‘2030 Megaproject’-New Generation Artificial Intelligence of China (2018AAA0100904), the National Key Research and Development Program of China (2022YFC3340900) and the National Social Science Foundation Major Project of China (20 & ZD047). L. van der Torre is financially supported by Luxembourg’s National Research Fund (FNR) through the project Deontic Logic for Epistemic Rights (OPEN O20/14776480) and the (Horizon 2020 funded) CHIST-ERA grant CHIST-ERA19-XAI (G.A. INTER/CHIST/19/14589586), as well as by the European Union’s Justice programme under grant agreement 101007420 (ADELE).

Commentary :

Accepted for publication with JAIR

Available on ORBilu :

since 29 December 2023

Statistics

Number of views

144 (2 by Unilu)

Number of downloads

80 (2 by Unilu)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenAlex citations

Bibliography

Alchourrón, C. E., & Bulygin, E. (1981). The expressive conception of norms. In Hilpinen, R. (Ed.), New studies in deontic logic, pp. 95–124. D. Reidel, Dordrecht.
Alchourrón, C. E. (1991). Conflicts of norms and revision of normative systems. Law and Philosophy, 10, 413–425.
Anderson, M., & Anderson, S. L. (2014). GenEth: A general ethical dilemma analyzer. In Proceedings of the 28th AAAI Conference on AI, pp. 253–261.
Arisaka, R., Dauphin, J., Satoh, K., & van der Torre, L. (2022). Multi-agent argumentation and dialogue. FLAP, 9(4), 891–924.
Arkin, R., Ulam, P., & Duncan, B. (2009). An ethical governor for constraining lethal action in an autonomous system. Tech. rep. GIT-GVU-09-02, Mobile Robot Laboratory, College of Computing, Georgia Tech.
Arkin, R., Ulam, P., & Wagner, A. R. (2012). Moral decision making in autonomous systems: enforcement, moral emotions, dignity, trust, and deception. Proc. of the IEEE, 100(3), 571–589.
Atkinson, K., & Bench-Capon, T. J. M. (2018). Taking account of the actions of others in value-based reasoning. Artif. Intell., 254, 1–20.
Awad, E., Dsouza, S., Kim, R., Schulz, J., Henrich, J., Shariff, A., Bonnefon, J., & Rahwan, I. (2018). The moral machine experiment. Nature, 563(7729), 59–64.
Awad, E., Anderson, M., Anderson, S. L., & Liao, B. (2020). An approach for combining ethical principles with public opinion to guide public policy. Artif. Intell., 287, 103349.
Baroni, P., Gabbay, D., Giacomin, M., & van der Torre, L. (Eds.). (2018). Handbook of Formal Argumentation, Vol. 1. College Publications.
Baroni, P., Giacomin, M., & Liao, B. (2015). Dealing with generic contrariness in structured argumentation. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, pp. 2727–2733.
Baroni, P., Giacomin, M., & Liao, B. (2018). A general semi-structured formalism for computational argumentation: Definition, properties, and examples of application. Artif. Intell., 257, 158–207.
Baum, S. D. (2020). Social choice ethics in artificial intelligence. AI Soc., 35(1), 165–176.
Bench-Capon, T. J. M., & Modgil, S. (2017). Norms and value based reasoning: justifying compliance and violation. Artif. Intell. Law, 25(1), 29–64.
Bench-Capon, T. J. M., Prakken, H., & Sartor, G. (2010). Argumentation in Legal Reasoning. Argumentation in Artificial Intelligence. Springer.
Biran, O., & Cotton, C. (2017). Explanation and justification in machine learning: A survey. In Proceedings of the IJCAI Workshop on Explainable Artificial Intelligence (XAI 2017), pp. 8–13.
Bjørgen, E. P., Madsen, S., Bjørknes, T. S., Heimsæter, F. V., Håvik, R., Linderud, M., Longberg, P., Dennis, L. A., & Slavkovik, M. (2018). Cake, death, and trolleys: Dilemmas as benchmarks of ethical decision-making. In Furman, J., Marchant, G. E., Price, H., & Rossi, F. (Eds.), Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, AIES 2018, New Orleans, LA, USA, February 02-03, 2018, pp. 23–29. ACM.
Boella, G., & van der Torre, L. (2006). Constitutive norms in the design of normative multiagent systems. In Computational Logic in Multi-Agent Systems, 6th International Workshop, CLIMA VI, LNCS 3900, pp. 303–319. Springer.
Booth, R., Caminada, M., & Marshall, B. (2018). DISCO: A web-based implementation of discussion games for grounded and preferred semantics. In Modgil, S., Budzynska, K., & Lawrence, J. (Eds.), Computational Models of Argument - Proceedings of COMMA 2018, Warsaw, Poland, 12-14 September 2018, Vol. 305 of Frontiers in Artificial Intelligence and Applications, pp. 453–454. IOS Press.
Brandt, F., Conitzer, V., Endriss, U., Lang, L., & Procaccia, A. D. (Eds.). (2016). Handbook of Computational Social Choice. Cambridge University Press.
Bremner, P., Dennis, L. A., Fisher, M., & Winfield, A. F. T. (2019). On proactive, transparent, and verifiable ethical reasoning for robots. Proc. IEEE, 107(3), 541–561.
Brewka, G. (1994). Reasoning about priorities in default logic. In Proceedings of the 12th National Conference on Artificial Intelligence, Seattle, WA, USA, July 31 - August 4, 1994, Volume 2., pp. 940–945.
Brewka, G., & Eiter, T. (1999). Preferred answer sets for extended logic programs. Artif. Intell., 109(1-2), 297–356.
Bringsjord, S., Arkoudas, K., & Bello, P. (2008). Toward a general logicist methodology for engineering ethically correct robots. IEEE Intelligent Systems, 21(4), 38–44.
Broersen, J. M., Dastani, M., Hulstijn, J., Huang, Z., & van der Torre, L. W. N. (2001). The BOID architecture: conflicts between beliefs, obligations, intentions and desires. In André, E., Sen, S., Frasson, C., & Müller, J. P. (Eds.), Proceedings of the Fifth International Conference on Autonomous Agents, AGENTS 2001, Montreal, Canada, May 28 - June 1, 2001, pp. 9–16. ACM.
Caminada, M., & Amgoud, L. (2007). On the evaluation of argumentation formalisms. Artif. Intell., 171(5-6), 286–310.
Charisi, V., Dennis, L., Fisher, M., Lieck, R., Matthias, A., Slavkovik, M., Sombetzki, J., Winfield, A., & Yampolskiy, R. (2017). Towards moral autonomous systems. CoRR, abs/1703.04741.
Chopra, A., van der Torre, L., Verhagen, H., & Villata, S. (2018). Handbook of normative multiagent systems. College Publications.
Cocarascu, O., Čyras, K., & Toni, F. (2018). Explanatory predictions with artificial neural networks and argumentation. In Proceedings of the IJCAI/ECAI Workshop on Explainable Artificial Intelligence (XAI 2018), pp. 26–32.
Cyras, K., & Toni, F. (2016). ABA+: assumption-based argumentation with preferences. In Baral, C., Delgrande, J. P., & Wolter, F. (Eds.), Principles of Knowledge Representation and Reasoning: Proceedings of the Fifteenth International Conference, KR 2016, Cape Town, South Africa, April 25-29, 2016, pp. 553–556. AAAI Press.
da Costa Pereira, C., Liao, B., Malerba, A., Rotolo, A., Tettamnzi, A. G. B., van der Torre, L., & Villata, S. (2018). Handling norms in multi-agent system by means of formal argumentation. In Chopra, A., van der Torre, L., Verhagen, H., & Villata, S. (Eds.), Handbook of Normative Multiagent Systems, pp. 345–373. College Publications.
Dennis, L. A., Fisher, M., Slavkovik, M., & Webster, M. P. (2016). Formal verification of ethical choices in autonomous systems. Robotics and Autonomous Systems, 77, 1–14.
Dignum, V. (2017). Responsible autonomy. In Proceedings of the 26th IIJCAI, pp. 4698–4704.
Dung, P. M. (1995). On the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games. Artificial Intelligence, 77(2), 321–358.
Dung, P. M., Thang, P. M., & Son, T. C. (2019). On structured argumentation with conditional preferences. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pp. 2792–2800. AAAI Press.
Dyrkolbotn, S., Pedersen, T., & Slavkovik, M. (2018). On the distinction between implicit and explicit ethical agency. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, AIES 2018, New Orleans, LA, USA, February 02-03, 2018, pp. 74–80.
Etzioni, A., & Etzioni, O. (2017). Incorporating ethics into artificial intelligence. The Journal of Ethics, 21(4), 403–418.
Fan, X., & Toni, F. (2015). On computing explanations in argumentation. In AAAI, pp. 1496–1502. AAAI Press.
Gabbay, D., Horty, J., Parent, X., van der Meyden, R., & van der Torre, L. (Eds.). (2013). Handbook of Deontic Logic and Normative Systems. College Publications, London.
Gabbay, D., Giacomin, M., Simari, G., & Thimm, M. (Eds.). (2021). Handbook of Formal Argumentation, Vol. 2. College Publications.
Gabbay, D. M., Giacomin, M., Liao, B., & van der Torre, L. W. N. (2018). Present and future of formal argumentation (dagstuhl perspectives workshop 15362). Dagstuhl Manifestos, 7(1), 69–95.
Gaggl, S. A., & Woltran, S. (2013). The cf2 argumentation semantics revisited. J. Log. Comput., 23(5), 925–949.
Grossi, D., & Jones, A. (2013). Constitutive norms and counts-as conditionals. In Horty, J., Gabbay, D., Parent, X., van der Meyden, R., & van der Torre, L. (Eds.), Handbook of Deontic Logic and Normative Systems, pp. 407–441. College Publications, London.
Hadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2017). The off-switch game. In The Workshops of the The Thirty-First AAAI Conference on Artificial Intelligence, Saturday, February 4-9, 2017, San Francisco, California, USA.
Horty, J. F. (1994). Moral dilemmas and nonmonotonic logic. J. Philos. Log., 23(1), 35–65.
Liao, B. S., Jin, L., & Koons, R. C. (2011). Dynamics of argumentation systems: A division-based method. Artif. Intell., 175(11), 1790–1814.
Liao, B., Anderson, M., & Anderson, S. L. (2021). Representation, justification, and explanation in a value-driven agent: an argumentation-based approach. AI Ethics, 1(1), 5–19.
Liao, B., Oren, N., van der Torre, L. W. N., & Villata, S. (2016). Prioritized norms and defaults in formal argumentation. In Roy, O., Tamminga, A. M., & Willer, M. (Eds.), Deontic Logic and Normative Systems - 13th International Conference, DEON 2016, Bayreuth, Germany, July 18-21, 2018, pp. 139–154. College Publications.
Liao, B., & van der Torre, L. (2020). Explanation semantics for abstract argumentation. In Prakken, H., Bistarelli, S., Santini, F., & Taticchi, C. (Eds.), Computational Models of Argument - Proceedings of COMMA 2020, Perugia, Italy, September 4-11, 2020, Vol. 326 of Frontiers in Artificial Intelligence and Applications, pp. 271–282. IOS Press.
Lindahl, L., & Odelstad, J. (2013). TJS. a formal framework for normative systems with intermediaries. In Horty, J., Gabbay, D., Parent, X., van der Meyden, R., & van der Torre, L. (Eds.), Handbook of Deontic Logic and Normative Systems. College Publications.
Lindner, F., & Bentzen, M. (2017). The hybrid ethical reasoning agent IMMANUEL. In Companion of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, HRI 2017, Vienna, Austria, March 6-9, 2017, pp. 187–188.
Makinson, D. (1999). On a fundamental problem of deontic logic. In McNamara, P., & Prakken, H. (Eds.), Norms, Logics and Information Systems. New Studies on Deontic Logic and Computer Science, pp. 29–54. IOS Press.
Makinson, D., & van der Torre, L. (2000). Input-output logics. Journal of Philosophical Logic, 29(4), 383–408.
Malle, B. F., Scheutz, M., Arnold, T., Voiklis, J., & Cusimano, C. (2015). Sacrifice one for the good of many?: People apply different moral norms to human and robot agents. In Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction, HRI’15, pp. 117–124. ACM.
Martino, A. A., & Socci Natali, F. (Eds.). (1986). Permissive Norms and Normative Concepts. Amsterdam: North Holland.
Miller, T. (2019). Explanation in artificial intelligence: Insights from the social sciences. Artif. Intell., 267, 1–38.
Modgil, S. (2009). Reasoning about preferences in argumentation frameworks. Artif. Intell., 173(9-10), 901–934.
Modgil, S., & Prakken, H. (2013). A general account of argumentation with preferences. Artif. Intell., 195, 361–397.
Moor, J. H. (2006). The nature, importance, and difficulty of machine ethics. IEEE Intelligent Systems, 21(4), 18–21.
Pardo, P., & Straßer, C. (2022). Modular orders on defaults in formal argumentation. Journal of Logic and Computation, 33, forthcoming.
Parent, X., & van der Torre, L. (2018). Introduction to Deontic Logic and Normative Systems. College Publications.
Pigozzi, G., & van der Torre, L. (2018). Arguing about constitutive and regulative norms. Journal of Applied Non-Classical Logics, 28(2-3), 189–217.
Reiter, R. (1980). A logic for default reasoning. Artif. Intell., 13(1-2), 81–132.
Robinson, P. (2021). Moral disagreement and artificial intelligence. In Fourcade, M., Kuipers, B., Lazar, S., & Mulligan, D. K. (Eds.), AIES’21: AAAI/ACM Conference on AI, Ethics, and Society, Virtual Event, USA, May 19-21, 2021, p. 209. ACM.
Ross, A. (1941). Imperatives and logic. Theoria, 7, 53–71. Reprinted in Philosophy of Science 11:30–46, 1944.
Ross, A. (1957). Tû-tû. Harvard Law Review, 70, 812–825.
Searle, J. (1969). Speech Acts. An Essay in the Philosophy of Language. Cambridge University Press, Cambridge.
Shariff, A., Bonnefon, J., & Rahwan, I. (2017). Psychological roadblocks to the adoption of self-driving vehicles. Nature Human Behaviour, 1(10), 694–696.
Sileno, G., Boer, A., & van Engers, T. (2014). Implementing explanation-based argumentation using answer set programming. In Eleventh International Workshop on Argumentation in Multi-Agent Systems (ARGMAS 2014): Paris, France, May 5, 2014. Cambridge, MA: Massachusetts Institute of Technology.
Stenius, E. (1963). Principles of a logic of normative systems. Acta Philosophica Fennica, 16, 247–260.
Vanderelst, D., & Winfield, A. F. T. (2018). An architecture for ethical robots inspired by the simulation theory of cognition. Cogn. Syst. Res., 48, 56–66.
Čyras, K., Satoh, K., & Toni, F. (2016). Explanation for case-based reasoning via abstract argumentation. In Computational Models of Argument - Proceedings of COMMA, pp. 243–254.
Vreeswijk, G., & Prakken, H. (2000). Credulous and sceptical argument games for preferred semantics. In JELIA, Vol. 1919 of LNCS, pp. 239–253. Springer.
Wallach, W., & Allen, C. (2008). Moral Machines: Teaching Robots Right from Wrong. Oxford University Press.
Walton, D. (2011). A dialogue system specification for explanation. Synthese, 182(3), 349–374.
Young, A. P., Modgil, S., & Rodrigues, O. (2016). Prioritised default logic as rational argumentation. In Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems, Singapore, May 9-13, 2016, pp. 626–634.