Artificial Intelligence-Enabled Interference Management for the Next Generation of Satellite Communications

The rapid deployment of ultra-dense non-geostationary satellite orbit (NGSO) constellations is fundamentally transforming satellite communication systems (SatComs) compared to legacy geostationary satellite orbit (GSO) architectures by enabling low-latency, high-capacity, and ubiquitous global connectivity. While these developments unlock new services and mission-critical applications worldwide, they also introduce complex interference challenges arising from spectrum scarcity, multi-constellation coexistence, and highly dynamic orbital motion. Conventional interference management techniques typically rely on accurate channel estimation and often fail in low signal-to-noise ratio (SNR) conditions and fast-varying NGSO environments, necessitating the adoption of more intelligent, adaptive, and data-driven solutions. This thesis investigates artificial intelligence (AI)–enabled interference management frameworks for next-generation SatComs, with a particular focus on downlink co-frequency interference (CFI) in multi-operator coexistence scenarios. Advanced AI-based interference detection and mitigation techniques are proposed to address the unique challenges imposed by NGSO system dynamics and large-scale deployments.

First, we tackled NGSO-to-GSO downlink interference detection at GSO ground stations (GGSs) by proposing advanced generative AI (GenAI)-based models, including variational autoencoders (VAEs) and transformer-based interference detector (TrID). These models are trained to generate samples of the expected interference-free GSO signal, then NGSO interference can be flagged by thresholding the generated samples' error with respect to the input signal. Realistic satellite characteristics, trajectories, and system parameters are used to emulate the interference scenarios and validate the proposed models. Numerical evaluation reveals that time-domain features exhibit higher accuracy for detecting interference compared to the frequency-domain features. Furthermore, the results demonstrate that TrID significantly outperforms other deep learning (DL)-based models as well as the traditional energy detector (ED) approach, showing an increase of up to 31.23% in interference detection accuracy.

We then introduce a novel attention-based blind adaptive beamforming model (AttBF) for NGSO-to-NGSO interference mitigation at NGSO user terminals (UTs). By leveraging estimation-free signal representations (e.g., received time-domain signals, frequency-domain representations, and sample covariance matrices (SCMs)), we evaluate the proposed AttBF model across various interference scenarios, encompassing both low spatial correlation (at UT's side-lobe) and high spatial correlation (at UT's main-lobe). Our findings highlight the enhanced spatial interference nulling capabilities of the AttBF-based approach, particularly when employing SCMs data, which offer near-optimal Signal-to-Interference-plus-Noise Ratio (SINR) performance with the fastest inference (~ 4.5 microseconds), compared to other DL-based beamforming models, such as convolutional neural networks (CNNs), and traditional methods, including zero forcing beamformer (ZFBF) and sample matrix inversion (SMI).

Finally, we propose a federated learning–enabled collaborative beamforming framework (FedSatBF) for NGSOs CFI interference mitigation in direct satellite-to-vehicle systems (DS2Vs). This work is also related to the previous second work by addressing the AttBF model generalization aspects and the various users' training capabilities. The proposed approach enables a serving satellite to aggregate locally trained attention-based beamforming models from a subset of vehicular user terminals (VUTs) using federated averaging, and broadcasts a global model that can be directly applied by both participating and non-participating VUTs experiencing similar CFI conditions. The proposed framework is explicitly designed to operate within the limited channel coherence time imposed by NGSO mobility. We investigate the impact of key FL hyperparameters, including per-coherent block communication rounds, per-round training data, per-VUT local epochs, VUT participation, and geographical VUT clustering, on convergence speed, interference nulling capability, and computational efficiency. Simulation results demonstrate near-optimal interference suppression performance, within 0.1 dB of optimal ZFBF, and strong generalization to non-participating users, highlighting the practicality of federated AI for large-scale NGSO deployments.

This thesis demonstrates that AI-driven interference management is a scalable and effective paradigm for enhancing the robustness, efficiency, and operational autonomy of next-generation satellite communication systems, while also contributing new open datasets enabled by realistic satellite orbit modeling.