Data pipelines; Image-based; Machine learning applications; Multirobots; Performance; Photo-realistic; Real-world; Robotic explorations; Robotic simulator; Synthetic data; Software; Control and Systems Engineering; Electrical and Electronic Engineering; Artificial Intelligence; Computer Science - Robotics; Computer Science - Computer Vision and Pattern Recognition
Abstract :
[en] Developing algorithms for extra-terrestrial robotic exploration has always been challenging. Along with the complexity associated with these environments, one of the main issues remains the evaluation of said algorithms. With the regained interest in lunar exploration, there is also a demand for quality simulators that will enable the development of lunar robots. In this paper, we propose Omniverse Lunar Robotic-Sim (OmniLRS) that is a photorealistic Lunar simulator based on Nvidia's robotic simulator. This simulation provides fast procedural environment generation, multi-robot capabilities, along with synthetic data pipeline for machine-learning applications. It comes with ROS1 and ROS2 bindings to control not only the robots, but also the environments. This work also performs sim-to-real rock instance segmentation to show the effectiveness of our simulator for image-based perception. Trained on our synthetic data, a yolov8 model achieves performance close to a model trained on real-world data, with 5% performance gap. When finetuned with real data, the model achieves 14% higher average precision than the model trained on real-world data, demonstrating our simulator's photorealism. The code is fully open-source, accessible here: https://github.com/AntoineRichard/OmniLRS, and comes with demonstrations.
Research center :
Interdisciplinary Centre for Security, Reliability and Trust (SnT) > SpaceR – Space Robotics
Disciplines :
Computer science
Author, co-author :
RICHARD, Antoine ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust > Space Robotics > Team Miguel Angel OLIVARES MENDEZ
KAMOHARA, Junnosuke ; University of Luxembourg ; Graduate School of Engineering, Tohoku University, Space Robotics Lab. (SRL), Department of Aerospace Engineering, Sendai, Japan
Uno, Kentaro; Graduate School of Engineering, Tohoku University, Space Robotics Lab. (SRL), Department of Aerospace Engineering, Sendai, Japan
Santra, Shreya; Graduate School of Engineering, Tohoku University, Space Robotics Lab. (SRL), Department of Aerospace Engineering, Sendai, Japan
VAN DER MEER, Dave ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > Space Robotics
OLIVARES MENDEZ, Miguel Angel ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > Space Robotics
Yoshida, Kazuya; Graduate School of Engineering, Tohoku University, Space Robotics Lab. (SRL), Department of Aerospace Engineering, Sendai, Japan
External co-authors :
yes
Language :
English
Title :
OmniLRS: A Photorealistic Simulator for Lunar Robotics
Publication date :
May 2024
Event name :
2024 IEEE International Conference on Robotics and Automation (ICRA)
Event place :
Yokohama, Japan
Event date :
13-05-2024 => 17-05-2024
Audience :
International
Main work title :
2024 IEEE International Conference on Robotics and Automation, ICRA 2024
Publisher :
Institute of Electrical and Electronics Engineers Inc., Piscataway, New Jersey, United States
E. Z. Crues, S. J. Lawrence, P. Bielski, A. B. Jacobs, J. Schlueter, A. Jagge, C. Foreman, C. Raymond, and N. Davis, "Digital lunar exploration sites (dles)," in 2022 IEEE Aerospace Conference (AERO), 2022, pp. 1-13.
L. Bingham, J. Kincaid, B. Weno, N. Davis, E. Paddock, and C. Foreman, "Digital lunar exploration sites unreal simulation tool (dust)," in 2023 IEEE Aerospace Conference, 2023, pp. 1-12.
M. Allan, U. Wong, P. M. Furlong, A. Rogg, S. McMichael, T. Welsh, I. Chen, S. Peters, B. Gerkey, M. Quigley, et al., "Planetary rover simulation for lunar exploration missions," in 2019 IEEE Aerospace Conference. IEEE, 2019, pp. 1-19.
I. Martin, M. Dunstan, and M. S. Gestido, "Planetary surface image generation for testing future space missions with pangu," in 2nd RPI Space Imaging Workshop. Sensing, Estimation, and Automation Laboratory, 2019.
N. Koenig and A. Howard, "Design and use paradigms for gazebo, an open-source multi-robot simulator," in 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566), vol. 3, 2004, pp. 2149-2154 vol.3.
A. Orsula, S. Bogh, M. Olivares-Mendez, and C. Martinez, "Learning to grasp on the moon from 3d octree observations with deep reinforcement learning," in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022, pp. 4112-4119.
C. Kilic, C. Tatsch, G. Pereira, J. Gross, et al., "Multi-robot cooperation for lunar in-situ resource utilization," Frontiers in Robotics and AI, vol. 10, 2023.
N. Khera, FauxRanger, 2020. [Online]. Available: https://dev.azure. com/nkhera/ git/FauxRanger
B. P. Dominik Pieczynski, LunarSim-ROS2-Connected Lunar Rover Simulation, 2023. [Online]. Available: https://github.com/PUTvision/ LunarSim
Z. Moratto, M. Broxton, R. A. Beyer, M. Lundy, and K. Husmann, "Ames stereo pipeline, nasa's open source automated stereogrammetry software," 2010. [Online]. Available: https://api.semanticscholar.org/ CorpusID:127931576
Pixar universal scene description," https://openusd.org/release/index.html, accessed: 2023-09-15.
G. Jocher, A. Chaurasia, and J. Qiu, "YOLO by Ultralytics," Jan. 2023. [Online]. Available: https://github.com/ultralytics/ultralytics
E. Games, Virtual Shadow Maps. [Online]. Available: https://docs. unrealengine.com/5.0/en-US/virtual-shadow-maps-in-unreal-engine/
A. Kar, A. Prakash, M.-Y. Liu, E. Cameracci, J. Yuan, M. Rusiniak, D. Acuna, A. Torralba, and S. Fidler, "Meta-sim: Learning to generate synthetic datasets," 2019.
J. Devaranjan, A. Kar, and S. Fidler, "Meta-sim2: Unsupervised learning of scene structure for synthetic data generation," 2020.
R. Pessia et al., 2020. [Online]. Available: https://www.kaggle.com/ datasets/romainpessia/artificial-lunar-rocky-landscape-dataset
W. Boerdijk et al., "Resyris-a real-synthetic rock instance segmentation dataset for training and benchmarking," in 2023 IEEE Aerospace Conference, 2023, pp. 1-9.
M. G. Müller et al., "A photorealistic terrain simulation pipeline for unstructured outdoor environments," in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021, pp. 9765-9772.
C. Wang, Z. Zhang, Y. Zhang, R. Tian, and M. Ding, "Gmsri: A texture-based martian surface rock image dataset," Sensors, vol. 21, no. 16, 2021. [Online]. Available: https://www.mdpi.com/1424-8220/ 21/16/5410
J. Zhang, L. Lin, Z. Fan, W. Wang, and J. Liu, "S5mars: Self-supervised and semi-supervised learning for mars segmentation," ArXiv, vol. abs/2207.01200, 2022. [Online]. Available: https: //api.semanticscholar.org/CorpusID:250264816
R. M. Swan et al., "Ai4mars: A dataset for terrain-aware autonomous driving on mars," in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2021, pp. 1982-1991.
B. Rothrock et al., "Spoc: Deep learning-based terrain classification for mars rover missions," 2016.
S. Coloma, C. Martinez, B. C. Yalcn, and M. A. Olivares-Mendez, "Enhancing rover teleoperation on the moon with proprioceptive sensors and machine learning techniques," IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 11434-11441, 2022.
I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial networks," 2014.
T. Park, M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu, "Semantic image synthesis with spatially-adaptive normalization," 2019.
P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, "Image-to-image translation with conditional adversarial networks," 2016.
M. Barker, E. Mazarico, G. Neumann, M. Zuber, J. Haruyama, and D. Smith, "A new lunar digital elevation model from the lunar orbiter laser altimeter and selene terrain camera," Icarus, vol. 273, pp. 346-355, 2016.
J. D. Stopar, M. S. Robinson, O. S. Barnouin, A. S. McEwen, E. J. Speyerer, M. R. Henriksen, and S. S. Sutton, "Relative depths of simple craters and the nature of the lunar regolith," Icarus, vol. 298, pp. 34-48, 2017, lunar Reconnaissance Orbiter-Part III. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S0019103517304013
E. Games, "Realitycapture," accessed: 2023-09-15. [Online]. Available: https://www.capturingreality.com/realitycapture
E. Blumenfeld, K. Beaulieu, A. Thomas, C. Evans, R. Zeigler, E. Oshel, D. Liddle, K. Righter, R. Hanna, and R. Ketcham, "3d virtual astromaterials samples collection of nasa's apollo lunar and antarctic meteorite samples to be an online database to serve researchers and the public," in 50th Lunar and Planetary Science Conference, held, 2019, pp. 18-22.
Leo rover," https://www.leorover.tech/the-rover, accessed: 2023-09-13.
D. Rodríguez-Martínez, K. Uno, K. Sawa, M. Uda, G. Kudo, G. H. Diaz, A. Umemura, S. Santra, and K. Yoshida, "Enabling faster locomotion of planetary rovers with a mechanically-hybrid suspension," arXiv preprint arXiv:2307.04494, 2023.
L. Keselman, J. I. Woodfill, A. Grunnet-Jepsen, and A. Bhowmik, "Intel realsense stereoscopic depth cameras," 2017.
J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, "Domain randomization for transferring deep neural networks from simulation to the real world," 2017.
E. Jones, "Apollo 17 image library," 2017. [Online]. Available: https://history.nasa.gov/alsj/a17/images17.html#MagC
T. Whitted, "An improved illumination model for shaded display," Commun. ACM, vol. 23, no. 6, p. 343-349, jun 1980. [Online]. Available: https://doi.org/10.1145/358876.358882
J. Tremblay, A. Prakash, D. Acuna, M. Brophy, V. Jampani, C. Anil, T. To, E. Cameracci, S. Boochoon, and S. Birchfield, "Training deep networks with synthetic data: Bridging the reality gap by domain randomization," 2018.
T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Dollár, "Microsoft coco: Common objects in context," 2014.
