[en] This paper presents an integrated Reinforcement Learning (RL) and Model Predictive Control (MPC) framework for autonomous satellite docking with a partially filled fuel tank. Traditional docking control faces challenges due to fuel sloshing in microgravity, which induces unpredictable forces affecting stability. To address this, we integrate Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) RL algorithms with MPC, leveraging MPC’s predictive capabilities to accelerate RL training and improve control robustness. The proposed approach is validated through Zero-G Lab of SnT experiments for planar stabilization and high-fidelity numerical simulations for 6-DOF docking with fuel sloshing dynamics. Simulation results demonstrate that SAC-MPC achieves superior docking accuracy, higher success rates, and lower control effort, outperforming standalone RL and PPO-MPC methods. This study advances fuel-efficient and disturbance-resilient satellite docking, enhancing the feasibility of on-orbit refueling and servicing missions.
B. C. Yalcin, C. Martinez Luna, S. Coloma Chacon, E. Skrzypczyk, and M. A. Olivares Mendez, "Ultra-Light Floating Platform: An Orbital Emulator for Space Applications, " presented at the IEEE International Conference on Robotics and Automation 2023 (ICRA), London, 30/5/2023, 2023.
N. Fries, P. Behruzi, T. Arndt, M. Winter, G. Netter, and U. Renner, "Modelling of fluid motion in spacecraft propellant tanks-sloshing, " in Space propulsion 2012 conference, 2012, pp. 89-94.
M. A. Ayoubi, F. A. Goodarzi, and A. Banerjee, "Attitude motion of a spinning spacecraft with fuel sloshing and nutation damping, " The Journal of the Astronautical Sciences, vol. 58, no. 4, pp. 551-568, 2011.
M. Deng and B. Yue, "Nonlinear model and attitude dynamics of flexible spacecraft with large amplitude slosh, " Acta Astronautica, vol. 133, pp. 111-120, 2017.
L. Mazmanyan and M. A. Ayoubi, "Fuzzy attitude control of spacecraft with fuel sloshing via linear matrix inequalities, " IEEE Transactions on Aerospace and Electronic Systems, vol. 54, no. 5, pp. 2526-2536, 2018.
M. Navabi, A. Davoodi, and M. Reyhanoglu, "Optimum fuzzy sliding mode control of fuel sloshing in a spacecraft using PSO algorithm, " Acta astronautica, vol. 167, pp. 331-342, 2020.
M. Navabi and A. Davoodi, "Fuzzy control of fuel sloshing in a spacecraft, " in 2018 6th Iranian Joint Congress on Fuzzy and Intelligent Systems (CFIS), 2018: IEEE, pp. 123-126.
J. E. Benmansour, R. Roubache, and B. Khouane, "Attitude Stabilization of a Flexible Satellite with Sloshing Phenomenon, " in 2023 International Conference on Networking and Advanced Systems (ICNAS), 2023: IEEE, pp. 1-5.
M. ALANDIHALLAJ, B. C. YALCIN, M. RAMEZANI, M. A. OLIVARES MENDEZ, J. THOEMEL, and A. HEIN, "Mitigating fuel sloshing disturbance in on-orbit satellite refueling: An experimental study, " in International Astronautical Congress IAC, 2023.
A. G. de Souza and L. C. de Souza, "Design of satellite attitude control system considering the interaction between fuel slosh and flexible dynamics during the system parameters estimation, " Applied Mechanics and Materials, vol. 706, pp. 14-24, 2015.
A. Bourdelle, J.-M. Biannic, H. Evain, S. Moreno, C. Pittet, and L. Burlion, "Propellant sloshing torque H?-based observer design for enhanced attitude control, " IFAC-PapersOnLine, vol. 52, no. 12, pp. 286-291, 2019.
M. Deng and B. Yue, "Attitude tracking control of flexible spacecraft with large amplitude slosh, " Acta Mechanica Sinica, vol. 33, pp. 1095-1102, 2017.
X. Song, Z. Fan, S. Lu, Y. Yan, and B. Yue, "Predefined-time sliding mode attitude control for liquid-filled spacecraft with large amplitude sloshing, " European Journal of Control, vol. 77, p. 100970, 2024.
M. A. Alandihallaj, A. M. Hein, and J. Thoemel, "Model predictive control-based satellite docking control for onorbit refueling mission, " Journal of Space Safety Engineering, 2025.
M. Ramezani, H. Habibi, and H. Voos, "UAV path planning employing MPC-reinforcement learning method considering collision avoidance, " arXiv preprint arXiv: 2302. 10669, 2023.
M. Ramezani, M. Amiri Atashgah, and A. Rezaee, "A Fault-Tolerant Multi-Agent Reinforcement Learning Framework for Unmanned Aerial Vehicles-Unmanned Ground Vehicle Coverage Path Planning, " Drones, vol. 8, no. 10, p. 537, 2024.
M. Ramezani and M. Amiri Atashgah, "Energy-aware hierarchical reinforcement learning based on the predictive energy consumption algorithm for search and rescue aerial robots in unknown environments, " Drones, vol. 8, no. 7, p. 283, 2024.
C. Mu, S. Liu, M. Lu, Z. Liu, L. Cui, and K. Wang, "Autonomous spacecraft collision avoidance with a variable number of space debris based on safe reinforcement learning, " Aerospace Science and Technology, vol. 149, p. 109131, 2024.
M. Ramezani, M. Alandihallaj, and A. M. Hein, "Fuel-Efficient and Fault-Tolerant CubeSat Orbit Correction via Machine Learning-Based Adaptive Control, " Aerospace (MDPI Publishing), vol. 11, no. 10, 2024.
M. RAMEZANI, M. Atashgah, M. ALANDIHALLAJ, and A. HEIN, "Reinforcement Learning for planning and task coordination in a swarm of CubeSats: Overcoming processor limitation challenges, " in International Astronautical Congress, 2023.
M. Ramezani, M. A. Alandihallaj, J. L. Sanchez-Lopez, and A. Hein, "Safe Hierarchical Reinforcement Learning for CubeSat Task Scheduling Based on Energy Consumption, " arXiv preprint arXiv: 2309. 12004, 2023.
G. Shetty, M. Ramezani, H. Habibi, H. Voos, and J. L. Sanchez-Lopez, "Motion Control in Multi-Rotor Aerial Robots Using Deep Reinforcement Learning, " in 2025 International Conference on Unmanned Aircraft Systems (ICUAS), 2025: IEEE, pp. 29-36.
M. A. Alandihallaj, M. Ramezani, and A. M. Hein, "MBSE-Enhanced LSTM Framework for Satellite System Reliability and Failure Prediction, " in MODELSWARD, 2024, pp. 349-356.
B. C. Yalçin, C. Martinez, S. Coloma, E. Skrzypczyk, and M. A. Olivares-Mendez, "Lightweight floating platform for ground-based emulation of on-orbit scenarios, " IEEE Access, vol. 11, pp. 94575-94588, 2023.
M. Olivares-Mendez et al., "Zero-G Lab: A multipurpose facility for emulating space operations, " Journal of Space Safety Engineering, vol. 10, no. 4, pp. 509-521, 2023.
M. Olivares-Mendez et al., "Establishing a multifunctional space operations emulation facility: Insights from the zero-g lab, " in Internal, 2023.
B. C. YALCIN, S. COLOMA CHACON, Z. BOKAL, C. MARTINEZ LUNA, and M. A. OLIVARES MENDEZ, "Pneumatic floating systems for performing zero-gravity experiments, " ed, 2024.
B. C. Yalçin, M. A. Alandihallaj, A. Hein, and M. Olivares-Mendez, "Advances in control techniques for floating platform stabilization in the zero-g lab, " in 17th Symposium on Advanced Space Technologies in Robotics and Automation (ASTRA), Leiden, The Netherlands, 2023.
H. Jasak, A. Jemcov, and Z. Tukovic, "OpenFOAM: A C++ library for complex physics simulations, " in International workshop on coupled methods in numerical dynamics, 2007, vol. 1000: Dubrovnik, Croatia), pp. 1-20.
M. A. Alandihallaj and M. R. Emami, "Satellite replacement and task reallocation for multiple-payload fractionated Earth observation mission, " Acta Astronautica, vol. 196, pp. 157-175, 2022.
M. A. A. Hallaj and N. Assadian, "Tethered satellite system control using electromagnetic forces and reaction wheels, " Acta Astronautica, vol. 117, pp. 390-401, 2015.
M. A. Alandihallaj, N. Assadian, and R. Varatharajoo, "Finite-time asteroid hovering via multiple-overlappinghorizon multiple-model predictive control, " Advances in Space Research, vol. 71, no. 1, pp. 645-653, 2023.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal policy optimization algorithms, " arXiv preprint arXiv: 1707. 06347, 2017.
T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, " in International conference on machine learning, 2018: PMLR, pp. 1861-1870.
M. Ramezani, M. A. Alandihallaj, and A. M. Hein, "PPO-Based Dynamic Control of Uncertain Floating Platforms in Zero-G Environment, " in 2024 IEEE International Conference on Robotics and Automation (ICRA), 2024: IEEE, pp. 11730-11736.
