Learning navigation attractors for mobile robots with reinforcement learning and reservoir computing

in Proceedings of the X Brazilian Congress on Computational Intelligence (CBIC) (2011)

Autonomous robot navigation in partially observable environments is a complex task because the state of the environment can not be completely determined only by the current sensory readings of a robot ... [more ▼]

Autonomous robot navigation in partially observable environments is a complex task because the state of the environment can not be completely determined only by the current sensory readings of a robot. This work uses the recently introduced paradigm for training recurrent neural networks (RNNs), called reservoir computing (RC), to model multiple navigation attractors in partially observable environments. In RC, the RNN with randomly generated fixed weights, called reservoir, projects the input into a high-dimensional dynamic space. Only the readout output layer is trained using standard linear regression techniques, and in this work, is used to approximate the state-action value function. By using a policy iteration framework, where an alternating sequence of policy improvement (samples generation from environment interaction) and policy evaluation (network training) steps are performed, the system is able to shape navigation attractors so that, after convergence, the robot follows the correct trajectory towards the goal. The experiments are accomplished using an e-puck robot extended with 8 distance sensors in a rectangular environment with an obstacle between the robot and the target region. The task is to reach the goal through the correct side of the environment, which is indicated by a temporary stimulus previously observed at the beginning of the episode. We show that the reservoir-based system (with short-term memory) can model these navigation attractors, whereas a feedforward network without memory fails to do so. [less ▲]

On Different Learning Approaches with Echo State Networks for Localization of Small Mobile Robots

in Proceedings of the IX Brazilian Conference on Neural Networks (2009)

Animals such as rats have innate and robust localization capabilities which allow them to navigate to goals in a maze. The rodent’s hippocampus, with the so called place cells, is responsible for such ... [more ▼]

Animals such as rats have innate and robust localization capabilities which allow them to navigate to goals in a maze. The rodent’s hippocampus, with the so called place cells, is responsible for such spatial processing. This work seeks to model these place cells using either supervised or unsupervised learning techniques. More specifically, we use a randomly generated recurrent neural network (the reservoir) as a non-linear temporal kernel to expand the input to a rich dynamic space. The reservoir states are linearly combined (using linear regression) or, in the unsupervised case, are used for extracting slowly-varying features from the input to form place cells (the architectures are organized in hierarchical layers). Experiments show that a small mobile robot with cheap and low-range distance sensors can learn to self-localize in its environment with the proposed systems. [less ▲]

Unsupervised Learning in Reservoir Computing: Modeling Hippocampal Place Cells for Small Mobile Robots

in Alippi, Cesare; Polycarpou, Marios; Panayiotou, Christos (Eds.) et al Artificial Neural Networks -- ICANN 2009 (2009)

Biological systems (e.g., rats) have efficient and robust localization abilities provided by the so called, place cells, which are found in the hippocampus of rodents and primates (these cells encode ... [more ▼]

Biological systems (e.g., rats) have efficient and robust localization abilities provided by the so called, place cells, which are found in the hippocampus of rodents and primates (these cells encode locations of the animal's environment). This work seeks to model these place cells by employing three (biologically plausible) techniques: Reservoir Computing (RC), Slow Feature Analysis (SFA), and Independent Component Analysis (ICA). The proposed architecture is composed of three layers, where the bottom layer is a dynamic reservoir of recurrent nodes with fixed weights. The upper layers (SFA and ICA) provides a self-organized formation of place cells, learned in an unsupervised way. Experiments show that a simulated mobile robot with 17 noisy short-range distance sensors is able to self-localize in its environment with the proposed architecture, forming a spatial representation which is dependent on the robot direction. [less ▲]

Modeling Multiple Autonomous Robot Behaviors and Behavior Switching with a Single Reservoir Computing Network

in Proceedings of the 2008 IEEE International Conference on Systems, Man and Cybernetics (2008)

Reservoir computing (RC) uses a randomly created Recurrent Neural Network as a reservoir of rich dynamics which projects the input to a high dimensional space. These projections are mapped to the desired ... [more ▼]

Reservoir computing (RC) uses a randomly created Recurrent Neural Network as a reservoir of rich dynamics which projects the input to a high dimensional space. These projections are mapped to the desired output using a linear output layer, which is the only part being trained by standard linear regression. In this work, RC is used for imitation learning of multiple behaviors which are generated by different controllers using an intelligent navigation system for mobile robots previously published in literature. Target seeking and exploration behaviors are conflicting behaviors which are modeled with a single RC network. The switching between the learned behaviors is implemented by an extra input which is able to change the dynamics of the reservoir, and in this way, change the behavior of the system. Experiments show the capabilities of Reservoir Computing for modeling multiple behaviors and behavior switching. [less ▲]

Mobile Robot Control in the Road Sign Problem using Reservoir Computing Networks

in Proceedings of the IEEE Int. Conf. on Robotics and Automation (ICRA) (2008)

In this work we tackle the road sign problem with reservoir computing (RC) networks. The T-maze task (a particular form of the road sign problem) consists of a robot in a T-shaped environment that must ... [more ▼]

In this work we tackle the road sign problem with reservoir computing (RC) networks. The T-maze task (a particular form of the road sign problem) consists of a robot in a T-shaped environment that must reach the correct goal (left or right arm of the T-maze) depending on a previously received input sign. It is a control task in which the delay period between the sign received and the required response (e.g., turn right or left) is a crucial factor. Delayed response tasks like this one form a temporal problem that can be handled very well by RC networks. Reservoir computing is a biologically plausible technique which overcomes the problems of previous algorithms such as backpropagation through time - which exhibits slow (or non-) convergence on training. RC is a new concept that includes a fast and efficient training algorithm. We show that this simple approach can solve the T-maze task efficiently. [less ▲]

Experiments with Reservoir Computing on the road sign problem

in Proceedings of the VIII Brazilian Congress on Neural Networks (CBRN) (2007)

The road sign problem is tackled in this work with Reservoir Computing (RC) networks. These networks are made of a fixed recurrent neural network where only a readout layer is trained. In the road sign ... [more ▼]

The road sign problem is tackled in this work with Reservoir Computing (RC) networks. These networks are made of a fixed recurrent neural network where only a readout layer is trained. In the road sign problem, an agent has to decide at some point in time which action to take given relevant information gathered in the past. We show that RC can handle simple and complex T-maze tasks (which are a subdomain of the road sign problem). [less ▲]

Generative Modeling of Autonomous Robots and their Environments using Reservoir Computing

in Neural Processing Letters (2007), 26(3), 233--249