[en] Tomorrow's robots will need to distinguish useful information from noise when performing different tasks. A household robot for instance may continuously
receive a plethora of information about the home, but needs to focus on just a
small subset to successfully execute its current chore. Filtering distracting
inputs that contain irrelevant data has received little attention in the
reinforcement learning literature. To start resolving this, we formulate a
problem setting in reinforcement learning called the $\textit{extremely noisy
environment}$ (ENE), where up to $99\%$ of the input features are pure noise.
Agents need to detect which features provide task-relevant information about
the state of the environment. Consequently, we propose a new method termed
$\textit{Automatic Noise Filtering}$ (ANF), which uses the principles of
dynamic sparse training in synergy with various deep reinforcement learning
algorithms. The sparse input layer learns to focus its connectivity on
task-relevant features, such that ANF-SAC and ANF-TD3 outperform standard SAC
and TD3 by a large margin, while using up to $95\%$ fewer weights. Furthermore,
we devise a transfer learning setting for ENEs, by permuting all features of
the environment after 1M timesteps to simulate the fact that other information
sources can become relevant as the world evolves. Again, ANF surpasses the
baselines in final performance and sample complexity. Our code is available at
https://github.com/bramgrooten/automatic-noise-filtering
Disciplines :
Computer science
Author, co-author :
Grooten, Bram; Eindhoven University of Technology [NL] > Mathematics and Computer Science
Sokar, Ghada; Eindhoven University of Technology [NL] > Mathematics and Computer Science
Dohare, Shibhansh; UAlberta - University of Alberta [CA] > Computer Science
Mocanu, Elena; University of Twente [NL] > Computer Science
Taylor, Matthew E.; UAlberta - University of Alberta [CA] > Computer Science
Pechenizkiy, Mykola; Eindhoven University of Technology [NL] > Mathematics and Computer Science
MOCANU, Decebal Constantin ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS) ; University of Twente [NL] > Computer Science
External co-authors :
yes
Language :
English
Title :
Automatic Noise Filtering with Dynamic Sparse Training in Deep Reinforcement Learning
Publication date :
30 May 2023
Event name :
AAMAS '23: 2023 International Conference on Autonomous Agents and Multiagent Systems
Event place :
London, United Kingdom
Event date :
from 29 May to 2 June 2023
Audience :
International
Main work title :
AAMAS '23: Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems
Publisher :
International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC
Kevin Acunto. 2012. Feature selection for scalable reinforcement learning. Ph.D. Dissertation. State University of New York at Binghamton. URL: https://www.proquest.com/openview/73f1a695503d3e32f0fa49cae57cb411. (Cited in Section 3)
Samin Yeasar Arnob, Riyasat Ohib, Sergey Plis, and Doina Precup. 2021. Single-Shot Pruning for Offline Reinforcement Learning. arXiv preprint arXiv:2112.15579 (2021). URL: https://arxiv.org/abs/2112.15579. (Cited in Section 3)
Zahra Atashgahi, Ghada Sokar, Tim van der Lee, Elena Mocanu, Decebal Constantin Mocanu, Raymond Veldhuis, and Mykola Pechenizkiy. 2022. Quick and Robust Feature Selection: the Strength of Energy-efficient Sparse Training for Autoencoders. Machine Learning 111, 1 (2022), 377-414. URL: https://link.springer.com/article/10.1007/s10994-021-06063-x. (Cited in Section 1, 3)
Paul Bach-y Rita, Carter C Collins, Frank A Saunders, Benjamin White, and Lawrence Scadden. 1969. Vision Substitution by Tactile Image Projection. Nature 221, 5184 (1969), 963-964. URL: https://www.nature.com/articles/221963a0. (Cited in Section 1)
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. 2018. Deep Rewiring: Training very sparse deep networks. International Conference on Learning Representations (2018). URL: https://arxiv.org/abs/1711.05136. (Cited in Section 3)
Wouter van den Bemd. 2022. Robust Deep Reinforcement Learning for Greenhouse Control and Crop Yield Optimization. Master's thesis. Eindhoven University of Technology. URL: https://research.tue.nl/en/studentTheses/robust-deepreinforcement-learning-for-greenhouse-control-and-cro. (Cited in Section 3)
Julian Bishop and Risto Miikkulainen. 2013. Evolutionary Feature Evaluation for Online Reinforcement Learning. In 2013 IEEE Conference on Computational Intelligence in Games (CIG). 1-8. https://doi.org/10.1109/CIG.2013.6633648 (Cited in Section 3)
Nicolò Botteghi, Khaled Alaa, Mannes Poel, Beril Sirmacek, Christoph Brune, Abeje Mersha, and Stefano Stramigioli. 2021. Low Dimensional State Representation Learning with Robotics Priors in Continuous Action Spaces. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 190-197. URL: https://arxiv.org/abs/2107.01667. (Cited in Section 3)
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. OpenAI Gym. arXiv preprint arXiv:1606.01540 (2016). URL: https://www.gymlibrary.dev/. (Cited in Section 1, 14)
Elisa Castaldi, Claudia Lunghi, and Maria Concetta Morrone. 2020. Neuroplasticity in adult human visual cortex. Neuroscience & Biobehavioral Reviews 112 (2020), 542-552. URL: https://www.sciencedirect.com/science/article/pii/S0149763419303288. (Cited in Section 1)
Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. 2021. Chasing Sparsity in Vision Transformers: An End-to-End Exploration. Advances in Neural Information Processing Systems 34 (2021). URL: https://arxiv.org/abs/2106.04533. (Cited in Section 3)
Tianlong Chen, Zhenyu Zhang, Pengjun Wang, Santosh Balachandra, Haoyu Ma, Zehao Wang, and Zhangyang Wang. 2022. Sparsity Winning Twice: Better Robust Generalization from More Efficient Training. International Conference on Machine Learning (2022). URL: https://openreview.net/forum?id=SYuJXrXq8tw. (Cited in Section 3)
Selima Curci, Decebal Constantin Mocanu, and Mykola Pechenizkiyi. 2021. Truly Sparse Neural Networks at Scale. arXiv preprint arXiv:2102.01732 (2021). URL: https://arxiv.org/abs/2102.01732. (Cited in Section C)
William Curran, Tim Brys, David Aha, Matthew Taylor, and William D Smart. 2016. Dimensionality Reduced Reinforcement Learning for Assistive Robots. In 2016 AAAI Fall Symposium Series. URL: https://www.aaai.org/ocs/index.php/FSS/FSS16/paper/download/14076/13660. (Cited in Section 3)
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. 2022. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature 602, 7897 (2022), 414-419. URL: https://www.nature.com/articles/s41586-021-04301-9. (Cited in Section 3)
Shibhansh Dohare, Richard S. Sutton, and A. Rupam Mahmood. 2021. Continual Backprop: Stochastic Gradient Descent with Persistent Randomness. arXiv preprint arXiv:2108.06325 (2021). URL: https://arxiv.org/abs/2108.06325. (Cited in Section 5)
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. 2020. Rigging the Lottery: Making All Tickets Winners. In International Conference on Machine Learning. PMLR, 2943-2952. URL: https://arxiv.org/abs/1911.11134. (Cited in Section 3)
Jonathan Frankle and Michael Carbin. 2019. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. International Conference on Learning Representations. URL: https://arxiv.org/abs/1803.03635. (Cited in Section 3)
Scott Fujimoto, Herke Hoof, and David Meger. 2018. Addressing Function Approximation Error in Actor-Critic Methods. In International Conference on Machine Learning. PMLR, 1587-1596. URL: https://arxiv.org/abs/1802.09477. (Cited in Section 1, 3, 4, 3)
Laura Graesser, Utku Evci, Erich Elsen, and Pablo Samuel Castro. 2022. The State of Sparse Training in Deep Reinforcement Learning. In International Conference on Machine Learning. PMLR, 7766-7792. URL: https://arxiv.org/abs/2206.10369. (Cited in Section 3, 4)
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In International Conference on Machine Learning. PMLR, 1861-1870. URL: https://arxiv.org/abs/1801.01290. (Cited in Section 1, 3, 4, A, 3)
Song Han, Jeff Pool, John Tran, and William Dally. 2015. Learning both Weights and Connections for Efficient Neural Networks. Advances in Neural Information Processing Systems 28 (2015). URL: https://arxiv.org/abs/1506.02626. (Cited in Section 3)
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. 2021. Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research 22, 241 (2021), 1-124. URL: https://arxiv.org/abs/2102.00554. (Cited in Section C)
Sara Hooker. 2021. The Hardware Lottery. Commun. ACM 64, 12 (2021), 58-65. URL: https://dl.acm.org/doi/10.1145/3467017. (Cited in Section C)
Diederik Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. International Conference for Learning Representations (2015). URL: https://arxiv.org/abs/1412.6980. (Cited in Section 4, 3)
Mark Kroon and Shimon Whiteson. 2009. Automatic Feature Selection for Model-Based Reinforcement Learning in Factored MDPs. In 2009 International Conference on Machine Learning and Applications. IEEE, 324-330. URL: https://ieeexplore.ieee.org/document/5381529. (Cited in Section 3)
Yuchen Li, Yifan Bao, Liyao Xiang, Junhan Liu, Cen Chen, Li Wang, and Xinbing Wang. 2021. Privacy Threats Analysis to Secure Federated Learning. arXiv preprint arXiv:2106.13076 (2021). URL: https://arxiv.org/abs/2106.13076. (Cited in Section 3)
Junjie Liu, Zhe Xu, Runbin Shi, Ray Cheung, and Hayden So. 2020. Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers. International Conference for Learning Representations (2020). URL: https://arxiv.org/abs/2005.06870. (Cited in Section 3)
Shiwei Liu, Decebal Constantin Mocanu, Amarsagar Reddy Ramapuram Matavalam, Yulong Pei, and Mykola Pechenizkiy. 2020. Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardware. Neural Computing and Applications 33 (2020), 2589-2604. URL: https://arxiv.org/abs/1901.09181. (Cited in Section C)
Shiwei Liu, Decebal Constantin Mocanu, and Mykola Pechenizkiy. 2019. On improving deep learning generalization with adaptive sparse connectivity. arXiv preprint arXiv:1906.11626 (2019). URL: https://arxiv.org/abs/1906.11626. (Cited in Section 3)
Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. 2021. Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training. In International Conference on Machine Learning. PMLR, 6989-7000. URL: https://arxiv.org/abs/2102.02887. (Cited in Section 3)
Decebal Constantin Mocanu, Elena Mocanu, Tiago Pinto, Selima Curci, Phuong H Nguyen, Madeleine Gibescu, Damien Ernst, and Zita A Vale. 2021. Sparse Training Theory for Scalable and Efficient Agents. Proceedings of the 20th International Conference on Autonomous Agents and Multiagent Systems (2021). URL: https://arxiv.org/abs/2103.01636. (Cited in Section 3, C)
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. 2018. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications 9, 1 (2018), 1-12. URL: https://arxiv.org/abs/1707.04780. (Cited in Section 1, 3, 4)
Janosch Moos, Kay Hansel, Hany Abdulsamad, Svenja Stark, Debora Clever, and Jan Peters. 2022. Robust Reinforcement Learning: A Review of Foundations and Recent Advances. Machine Learning and Knowledge Extraction 4, 1 (2022), 276-315. URL: https://www.mdpi.com/2504-4990/4/1/13. (Cited in Section 3)
Alberto E Pereda. 2014. Electrical synapses and their functional interactions with chemical synapses. Nature Reviews Neuroscience 15, 4 (2014), 250-263. URL: https://www.nature.com/articles/nrn3708. (Cited in Section 1)
Banafsheh Rafiee, Zaheer Abbas, Sina Ghiassian, Raksha Kumaraswamy, Richard S Sutton, Elliot A Ludvig, and Adam White. 2020. From Eye-blinks to State Construction: Diagnostic Benchmarks for Online Representation Learning. Adaptive Behavior (2020). URL: https://arxiv.org/abs/2011.04590. (Cited in Section 3)
David Silver, Satinder Singh, Doina Precup, and Richard S Sutton. 2021. Reward is enough. Artificial Intelligence 299 (2021), 103535. URL: https://www.sciencedirect.com/science/article/pii/S0004370221000862. (Cited in Section 3)
Ghada Sokar, Zahra Atashgahi, Mykola Pechenizkiy, and Decebal Constantin Mocanu. 2022. Where to Pay Attention in Sparse Training for Feature Selection?. In Advances in Neural Information Processing Systems. URL: https://openreview.net/forum?id=xWvI9z37Xd. (Cited in Section 1)
Ghada Sokar, Elena Mocanu, Decebal Constantin Mocanu, Mykola Pechenizkiy, and Peter Stone. 2022. Dynamic Sparse Training for Deep Reinforcement Learning. The 31st International Joint Conference on Artificial Intelligence (2022). URL: https://arxiv.org/abs/2106.04217. (Cited in Section 1, 3, 4, 9, A, C)
Austin Stone, Oscar Ramirez, Kurt Konolige, and Rico Jonschkowski. 2021. The Distracting Control Suite - A Challenging Benchmark for Reinforcement Learning from Pixels. arXiv preprint arXiv:2101.02722 (2021). URL: https://arxiv.org/abs/2101.02722. (Cited in Section 3)
Ke Sun, Yi Liu, Yingnan Zhao, Hengshuai Yao, Shangling Jui, and Linglong Kong. 2021. Exploring the Training Robustness of Distributional Reinforcement Learning against Noisy State Observations. arXiv preprint arXiv:2109.08776 (2021). URL: https://arxiv.org/abs/2109.08776. (Cited in Section 3)
Yiqin Tan, Pihe Hu, Ling Pan, and Longbo Huang. 2023. RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratch. International Conference on Learning Representations (2023). URL: https://arxiv.org/abs/2205.15043. (Cited in Section 3)
Matthew E Taylor and Peter Stone. 2009. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research 10, 7 (2009). URL: https://www.jmlr.org/papers/v10/taylor09a.html. (Cited in Section 3)
Emanuel Todorov, Tom Erez, and Yuval Tassa. 2012. MuJoCo: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 5026-5033. URL: https://mujoco.org/. (Cited in Section 1, 14)
Eugene Vinitsky, Yuqing Du, Kanaad Parvate, Kathy Jang, Pieter Abbeel, and Alexandre Bayen. 2020. Robust Reinforcement Learning using Adversarial Populations. arXiv preprint arXiv:2008.01825 (2020). URL: https://arxiv.org/abs/2008. 01825. (Cited in Section 3)
Marc Aurel Vischer, Robert Tjarko Lange, and Henning Sprekeler. 2022. On Lottery Tickets and Minimal Task Representations in Deep Reinforcement Learning. ICLR (2022). URL: https://arxiv.org/abs/2105.01648. (Cited in Section 1, 4, D.1)
Shimon Whiteson, Peter Stone, Kenneth O Stanley, Risto Miikkulainen, and Nate Kohl. 2005. Automatic Feature Selection in Neuroevolution. In Proceedings of the 7th annual conference on Genetic and evolutionary computation. 1225-1232. URL: https://dl.acm.org/doi/10.1145/1068009.1068210. (Cited in Section 3)
Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. 2020. Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch. International Conference on Learning Representations (2020). URL: https://arxiv.org/abs/2102.04010. (Cited in Section C)
