Reinforcement learning; Communication Theory; Multi-agent systems
Abstract :
[en] Consider a collaborative task carried out by two autonomous agents that can communicate over a noisy channel. Each agent is only aware of its own state, while the accomplishment of the task depends on the value of the joint state of both agents. As an example, both agents must simultaneously reach a certain location of the environment, while only being aware of their own positions. Assuming the presence of feedback in the form of a common reward to the agents, a conventional approach would apply separately: (\emph{i}) an off-the-shelf coding and decoding scheme in order to enhance the reliability of the communication of the state of one agent to the other; and (\emph{ii}) a standard multiagent reinforcement learning strategy to learn how to act in the resulting environment. In this work, it is argued that the performance of the collaborative task can be improved if the agents learn how to jointly communicate and act. In particular, numerical results for a baseline grid world example demonstrate that the jointly learned policy carries out compression and unequal error protection by leveraging information about the action policy.
Research center :
Interdisciplinary Centre for Security, Reliability and Trust (SnT) > Applied Security and Information Assurance Group (APSIA)
Disciplines :
Electrical & electronics engineering
Author, co-author :
Mostaani, Arsham ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT)
Simeone, Osvaldo; King's College London > Informatics
Chatzinotas, Symeon ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT)
Ottersten, Björn ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT)
External co-authors :
yes
Language :
English
Title :
Learning-based Physical Layer Communications for Multiagent Collaboration
Publication date :
11 September 2019
Event name :
IEEE 30th Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC)
D. V. Pynadath and M. Tambe, "The communicative multiagent team decision problem: Analyzing teamwork theories and models, " Journal of Artificial Intelligence Research, vol. 16, pp. 389-423, Jun. 2002.
G. Weiss, Multiagent systems: A modern approach to distributed artificial intelligence, MIT press, 1999.
M. Tan, "Multi-agent reinforcement learning: Independent vs. cooperative agents, " in Proc. International Conference on Machine Learning, San Francisco, CA, USA, 1998, pp. 487-494, Morgan Kaufmann Publishers Inc.
L. Busoniu, R. Babuska, and B. D. Schutter, "A comprehensive survey of multiagent reinforcement learning, " IEEE Trans. Systems, Man, and Cybernetics, Part C, vol. 38, no. 2, pp. 156-172, Mar. 2008.
M. Lauer and M. A. Riedmiller, "An algorithm for distributed reinforcement learning in cooperative multi-agent systems, " in Proc. Conference on Machine Learning. 2000, pp. 535-542, Morgan Kaufmann Publishers Inc.
A. Sahai and P. Grover, "Demystifying the witsenhausen counterexample [ask the experts], " IEEE Control Systems Magazine, vol. 30, no. 6, pp. 20-24, Dec. 2010.
F. Fischer, M. Rovatsos, and G. Weiss, "Hierarchical reinforcement learning in communication-mediated multiagent coordination, " in Proc. IEEE Joint Conference on Autonomous Agents and Multiagent Systems, 2004. AAMAS 2004., New York, Jul. 2004, pp. 1334-1335.
S. Sukhbaatar, R. Fergus, et al., "Learning multiagent communication with backpropagation, " in Proc. Advances in Neural Information Processing Systems, Barcelona, 2016, pp. 2244-2252.
J. Foerster, Y. Assael, N. D. Freitas, and Sh. Whiteson, "Learning to communicate with deep multi-agent reinforcement learning, " in Proc. Advances in Neural Information Processing Systems, Barcelona, 2016, pp. 2137-2145.
Q. Huang, E. Uchibe, and K. Doya, "Emergence of communication among reinforcement learning agents under coordination environment, " in Proc. IEEE Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob), Sept. 2016, pp. 57-58.
J. Foerster, N. Nardelli, G. Farquhar, Ph. Torr, P. Kohli, and Sh. Whiteson, "Stabilising experience replay for deep multi-agent reinforcement learning, " arXiv preprint arXiv: 1702. 08887, 2017.
R. Lowe, L. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch, "Multi-agent actor-critic for mixed cooperative-competitive environments, " in Proc. Advances in Neural Information Processing Systems, Long Beach, 2017, pp. 6382-6393.
J. N. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and Sh. Whiteson, "Counterfactual multi-agent policy gradients, " arXiv preprint arXiv: 1705. 08926, 2017.
Osvaldo Simeone, "A very brief introduction to machine learning with applications to communication systems, " IEEE Transactions on Cognitive Communications and Networking, vol. 4, no. 4, pp. 648-664, 2018.
Daewoo Kim, Sangwoo Moon, David Hostallero, Wan Ju Kang, Taeyoung Lee, Kyunghwan Son, and Yung Yi, "Learning to schedule communication in multi-agent reinforcement learning, " arXiv preprint arXiv: 1902. 01554, 2019.
R. Sutton and A. G. Barto, Introduction to reinforcement learning, vol. 135, MIT Press, 2 edition, Nov. 2017.
P. Xuan, V. Lesser, and Sh. Zilberstein, "Communication decisions in multi-agent cooperation: Model and experiments, " in Proc. ACM Conference on Autonomous Agents, 2001, pp. 616-623.
Ryan Lowe, Jakob Foerster, Y-Lan Boureau, Joelle Pineau, and Yann Dauphin, "On the pitfalls of measuring emergent communication, " arXiv preprint arXiv: 1903. 05168, 2019.