Extended Reality; Human-Robot Interaction; Space Robotics; Telerobotics
Abstract :
[en] The Moon is once again a focal point for space exploration, with interest extending from space agencies to private companies. Recent discoveries of lunar resources and the potential to develop a lunar economy based on in-Situ Resource Utilisation (ISRU) have accelerated the development of advanced robotic technologies. This is clear in the increasing number of missions, with three landers deployed in 2024, and an additional three as of March 2025. To ensure successful robotic lunar operations, reliable human supervision and control are essential, supported by advanced systems that enhance operator situational awareness. This paper presents the eXtended Reality interface for lunar ROVer Mission Control (XROV) - a novel system designed to enable immersive teleoperation and monitoring of lunar rovers. XROV provides operators with an online 3D virtual reconstruction of the rover’s lunar surroundings using on-board sensor data, allowing them to visualize regions of interest from multiple perspectives and identify risks such as wheel-ground interactions and hazards with Artificial Intelligence (AI). Additionally, a Graphical User Interface (GUI) is proposed to improve human-robot interaction, displaying critical information such as rock coordinates, battery levels, and terrain elevation. XROV employs a human-in-the-loop approach, which is essential for ensuring adaptability and informed decision making during AI-assisted rover missions. The system was designed, implemented, and tested at the Lunalab, University of Luxembourg, where two experiments were conducted with different operators and real rovers. The results demonstrate the effectiveness of XROV in improving situational awareness, control, and user experience, making it a valuable tool for teleoperating rovers in hazardous and unexplored lunar environments.
Disciplines :
Computer science
Author, co-author :
FRANTZ, Alexandre ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > Space Robotics
Coloma, Sofía ; Space Robotics (SpaceR) Research Group at the Interdisciplinary Research Centre for Security, Reliability and Trust (SnT), University of Luxembourg, Esch-sur-Alzette, Luxembourg
Olivares-Mendez, Miguel A. ; Space Robotics (SpaceR) Research Group at the Interdisciplinary Research Centre for Security, Reliability and Trust (SnT), University of Luxembourg, Esch-sur-Alzette, Luxembourg
External co-authors :
no
Language :
English
Title :
XROV: An Immersive eXtended Reality (XR) Interface for Lunar Rover Mission Control
Publication date :
2026
Event name :
XR Salento: International Conference on Extended Reality
Event place :
Otranto, Italy
Event date :
17-06-2025 => 20-06-2025
Audience :
International
Main work title :
Extended Reality - International Conference, XR Salento 2025, Proceedings
Editor :
De Paolis, Lucio Tommaso
Publisher :
Springer Science and Business Media Deutschland GmbH
Aymeric, T., Cesari, U., Foing, B.: Robotics, stereo vision and lidar for lunar exploration. Europlanet Science Congress 2020 (2020)
Bakambu, J., Gemme, S., Allard, P., Lamarche, T., Rekleitis, I., Dupius, E.: 3D reconstruction of environments for tele-operation of planetary rover (2006)
Bausys, R., Cavallaro, F., Semenas, R.: Application of sustainability principles for harsh environment exploration by autonomous robot. Sustainability 11(9), 2518 (2019)
Biggie, H., et al.: Flexible supervised autonomy for exploration in subterranean environments. arXiv preprint arXiv:2301.00771 (2023)
Burns, J.O., et al.: Science on the lunar surface facilitated by low latency telerobotics from a lunar orbital platform-gateway. Acta Astronaut. 154, 195–203 (2019)
Cardenas, I.S., et al.: AARON: assistive augmented reality operations and navigation system for NASA’s exploration extravehicular mobility unit (XEMU). In: Intelligent Human Computer Interaction: 12th International Conference, IHCI 2020, Daegu, South Korea, 24–26 November 2020, Proceedings, Part II 12, pp. 406–422. Springer (2021)
Chen, Y., Zhang, B., Zhou, J., Wang, K.: Real-time 3D unstructured environment reconstruction utilizing VR and kinect-based immersive teleoperation for agricultural field robots. Comput. Electron. Agric. 175, 105579 (2020)
Chen, Z., Wu, B., Liu, W.C.: Mars3DNet: CNN-based high-resolution 3D reconstruction of the Martian surface from single images. Remote Sens. 13(5), 839 (2021)
Coloma, S., Frantz, A., Van Der Meer, D., Skrzypczyk, E., Orsula, A., Olivares-Mendez, M.: Immersive rover control and obstacle detection based on extended reality and artificial intelligence. In: 2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pp. 743–744 (2024)
Coloma, S., Martinez, C., Yalçın, B.C., Olivares-Mendez, M.A.: Enhancing rover teleoperation on the moon with proprioceptive sensors and machine learning techniques. IEEE Robot. Autom. Lett. 7(4), 11434–11441 (2022)
Creech, S., Guidi, J., Elburn, D.: Artemis: an overview of NASA’s activities to return humans to the moon. In: 2022 IEEE Aerospace Conference (AERO), pp. 1– 7. IEEE (2022)
Crues, E.Z.: Returning to the moon: Plans for distributed simulation in the artemis program. In: IEEE/ACM DS-RT 2021-25th International Symposium on Distributed Simulation and Real Time Applications (2021)
Dobynde, M.I., Guo, J.: Radiation environment at the surface and subsurface of the moon: model development and validation. J. Geophys. Res. Planets 126(11), e2021JE006930 (2021)
European Space Agency (ESA): A new dimension in robotics. https://www.esa. int/Science Exploration/Human and Robotic Exploration/A new dimension in robotics. Accessed 10 Mar 2025
European Space Agency (ESA): Lunar rover driving in VR (2020). https://www. esa.int/ESA Multimedia/Videos/2020/04/Lunar rover driving in VR. Accessed 10 Mar 2025
European Space Agency (ESA): Extended Reality Lab for Mars Experiments (Mars XLab) (2022). https://www.esa.int/ESA Multimedia/Images/2022/06/Extended reality lab for Mars experiments Mars Xlab. Accessed 10 Mar 2025
European Space Agency (ESA): X-aRm (2022). https://www.esa.int/ESA Multimedia/Images/2022/06/X-aRm. Accessed 10 Mar 2025
Forsey-Smerek, A., et al.: Assessment of depth data acquisition methods for virtual reality mission operations support tools. In: 2022 IEEE Aerospace Conference (AERO), pp. 1–14. IEEE (2022)
González, C., Solanes, J.E., Munoz, A., Gracia, L., Girbés-Juan, V., Tornero, J.: Advanced teleoperation and control system for industrial robots based on augmented virtuality and haptic feedback. J. Manuf. Syst. 59, 283–298 (2021)
Hart, S.G.: Nasa-task load index (NASA-TLX); 20 years later. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 50, pp. 904–908. Sage Publications Sage, Los Angeles (2006)
hyounesy: ROS-TCP connector. https://github.com/Unity-Technologies/ROS-TCP-Connector. Accessed 10 Mar 2025
Jocher, G., et al.: ultralytics/YOLOv5: v7. 0-YOLOv5 SOTA realtime instance segmentation. Zenodo (2022)
Kaarlela, T., Padrao, P., Pitkäaho, T., Pieskä, S., Bobadilla, L.: Digital twins utilizing XR-technology as robotic training tools. Machines 11(1), 13 (2022)
Kuutti, S., Fallah, S., Katsaros, K., Dianati, M., Mccullough, F., Mouzakitis, A.: A survey of the state-of-the-art localization techniques and their potentials for autonomous vehicle applications. IEEE Internet Things J. 5(2), 829–846 (2018)
Labbé, M.: RTAB-Map. https://introlab.github.io/rtabmap/. Accessed 10 Mar 2025
Lino, G.J., Palacios, P., Napán, J.L., Cornejo, J.: Conceptual design of ARIES rover for water production and resource utilization on the moon. In: 2020 IEEE Engineering International Research Conference (EIRCON), pp. 1–4. IEEE (2020)
Lunghi, G., Marín Prades, R., Di Castro, M., Masi, A., Sanz, P.: Multimodal human-robot interface for accessible remote robotic interventions in hazardous environments. IEEE Access PP, 1 (2019)
McHenry, N., et al.: Predictive XR telepresence for robotic operations in space. In: 2021 IEEE Aerospace Conference (50100), pp. 1–10. IEEE (2021)
NASA: Nine ways we use AR and VR on the international space station. https://www.nasa.gov/mission pages/station/research/news/nine-ways-we-use-ar-vr-on-iss. Accessed 10 Mar 2025
National Aeronautics and Space Administration (NASA): NASA’s Lunar Exploration Program Overview (2020). https://www.nasa.gov/wp-content/uploads/2020/12/artemis plan-20200921.pdf?emrc=f43185. Accessed 10 Mar 2025
NDTV: NASA training space robots with virtual reality. https://www.ndtv.com/world-news/nasa-training-space-robots-with-virtual-reality-1257374. Accessed 10 Mar 2025
Panzirsch, M., Weber, B., Rubio, L., Coloma, S., Ferre, M., Artigas, J.: Tele-healthcare with humanoid robots: a user study on the evaluation of force feedback effects. In: 2017 IEEE World Haptics Conference (WHC), pp. 245–250. IEEE (2017)
Pryor, W., et al.: A virtual reality planning environment for high-risk, high-latency teleoperation. In: 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 11619–11625. IEEE (2023)
Roldao, L., Charette, R., Verroust, A.: 3D surface reconstruction from voxel-based lidar data, pp. 2681–2686 (2019)
Roldán Gómez, J., Peña-Tapia, E., García Auñón, P., Cerro, J., Barrientos, A.: Bringing adaptive and immersive interfaces to real-world multi-robot scenarios: application to surveillance and intervention in infrastructures. IEEE Access PP, 1 (2019)
ROS-Wiki: RTAB-Map ROS. http://wiki.ros.org/rtabmap ros. Accessed 10 Mar 2025
Sachdeva, R., et al.: Autonomy and perception for space mining. In: 2022 International Conference on Robotics and Automation (ICRA), pp. 4087–4093. IEEE (2022)
Seedhouse, E.L., Llanos, P.: Science and exploration of the moon enabled by surface telerobotics. J. Space Safety Eng. 8(3), 231–237 (2021)
Sharma, U.M., Chadha, R.: Exploring the applications and advances in 3D imaging and 3d reconstruction. In: 2023 International Conference on IoT, Communication and Automation Technology (ICICAT), pp. 1–6. IEEE (2023)
Shewchuk, J.: A two-dimensional quality mesh generator and Delaunay triangulator. https://www.cs.cmu.edu/∼quake/triangle.html. Accessed 10 Mar 2025
Smith, M., et al.: The Artemis program: an overview of NASA’s activities to return humans to the moon. In: 2020 IEEE Aerospace Conference, pp. 1–10. IEEE (2020)
Sun, D., Kiselev, A., Liao, Q., Stoyanov, T., Loutfi, A.: A new mixed-reality-based teleoperation system for telepresence and maneuverability enhancement. IEEE Trans. Hum.-Mach. Syst. 50(1), 55–67 (2020)
Szczurek, K., Marín Prades, R., Matheson, E., Rodriguez-Nogueira, J., Di Castro, M.: Multimodal multi-user mixed reality human–robot interface for remote operations in hazardous environments. IEEE Access PP, 1 (2023)
Szczurek, K.A., Prades, R.M., Matheson, E., Rodriguez-Nogueira, J., Di Castro, M.: Mixed reality human-robot interface with adaptive communications congestion control for the teleoperation of mobile redundant manipulators in hazardous environments. IEEE Access 10, 87182–87216 (2022)
Unity3D: Unity asset store. https://assetstore.unity.com/. Accessed 20 July 2023
Unity3D-Documentation: Unity scripting-mesh API. https://docs.unity3d.com/ScriptReference/Mesh.html. Accessed10 Mar 2025
Van Der Meer, D., et al.: REALMS: resilient exploration and lunar mapping system. Front. Robot. AI 10, 1127496 (2023)
Walker, M., et al.: A mixed reality supervision and telepresence interface for outdoor field robotics. In: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2345–2352. IEEE (2021)
Weller, R., Schröder, C., Teuber, J., Dittmann, P., Zachmann, G.: VR-interactions for planning planetary swarm exploration missions in VaMEx-Vtb. In: 2021 IEEE Aerospace Conference (50100), pp. 1–11. IEEE (2021)
Xia, Z., Bu, S., Chen, L., Han, P.: A fast reconstruction method based on real-time terrain following, pp. 1479–1484 (2022)
Xu, X., Fu, X., Zhao, H., Liu, M., Xu, A., Ma, Y.: Three-dimensional reconstruction and geometric morphology analysis of lunar small craters within the patrol range of the YUTU-2 rover. Remote Sens. 15 (2023)
Xu, Y., Stilla, U.: Toward building and civil infrastructure reconstruction from point clouds: a review on data and key techniques. IEEE J. Sel. Top. Appl. Earth Observ. Remote Sens. 14, 2857–2885 (2021)
Zarei, M., Chhabra, R.: Advancements in autonomous mobility of planetary wheeled mobile robots: a review. Front. Space Technol. 3, 1080291 (2022)
Zhong, J., Yan, J., Li, M., Barriot, J.P.: ISPRS journal of photogrammetry and remote sensing a deep learning-based local feature extraction method for improved image matching and surface reconstruction from YUTU-2 PCAM images on the moon. ISPRS J. Photogram. Remote Sens. 206, 16–29 (2023)
Zhou, Y., Li, X., Hua, B.: Crater identification simulation using lidar on lunar rover. Measurement 210, 112550 (2023)
