Reinforcement Learning for Attitude Control of a Spacecraft with Flexible Appendages

This study explores the reinforcement learning (RL) approach to constructing attitude control strategies for a LEOsatellite with flexible appendages. Attitude control system actuated by a set of three reaction wheels is considered.The satellite is assumed to move in a circular low Earth orbit under the action of gravity-gradient torque, randomdisturbance torque, and oscillations excited in flexible appendages. The control policy for rest-to-rest slew maneuversis learned via the Proximal Policy Optimization (PPO) technique. The robustness of the obtained control policy isanalyzed and compared to that of conventional controllers. The first part of the study is focused on problem formulationin terms of Markov Decision Processes, analysis of different reward-shaping techniques, and finally training the RL-agent and comparing the obtained results with the state-of-the-art RL-controllers as well as with the performance ofa commonly used quaternion feedback regulator (Lyapunov-based PD controller). We then proceed to consider thesame spacecraft with flexible appendages added to its structure. Equations of excitable oscillations are appended tothe system and coupling terms are added describing the interactions between the main rigid body and the flexiblestructures. The dynamics of the rigid spacecraft thus becomes coupled with that of its flexible appendages and thecontrol strategy should change accordingly in order to prevent actions that entail excitation of oscillation modes.Again PPO is used to learn the control policy for rest-to-rest slew maneuvers in the extended system. All in all,the proposed reinforcement learning strategy is shown to converge to a policy that matches the performance of thequaternion feedback regulator for a rigid spacecraft. It is also shown that a policy can be trained to take into accountthe highly nonlinear dynamics caused by the presence of flexible elements that need to be brought to rest in the requiredattitude. We also discuss the advantages of the reinforcement learning approach such as robustness and ability of onlinelearning pertaining to the systems that require a high level of autonomy