Energy efficiency; URLLC; Grant-free NOMA; Multi-agent deep reinforcement learning
Abstract :
[en] Grant-free non-orthogonal multiple access (GF-NOMA) has emerged as a promising access technology for the fifth generation and beyond wireless networks that enable ultra-reliable and low-latency communications (URLLC) to ensure low access latency and high connectivity density. Furthermore, designing energy-efficient (EE) resource allocation strategies is a crucial aspect of future cellular system development. Taking these goals into account, this paper proposes an EE sub-channel and power allocation strategy for URLLC-enabled GF-NOMA (URLLC-GF-NOMA) systems based on multi-agent (MA) deep reinforcement learning (MADRL). In particular, the URLLC-GF-NOMA methods using MA dueling double deep Q network (MA3DQN), MA double deep Q network (MA2DQN), and MA deep Q network (MADQN) techniques are designed to enable users to select the most appropriate sub-channel and transmission power for their communications. The aim is to build an efficient MADRL-based solution, ensuring rapid convergence with small signaling overhead, to maximize the network EE while fulfilling the URLLC requirements of all users. Simulation results show that the MADQN and MA2DQN methods, which have lower complexity than MA3DQN, are more appropriate for the URLLC-GF-NOMA systems under consideration. Moreover, our proposed methods exhibit superior convergence characteristics, a reduction in signaling overhead, and enhanced EE performance compared to other benchmark strategies.
Research center :
Interdisciplinary Centre for Security, Reliability and Trust (SnT) > SIGCOM - Signal Processing & Communications
Disciplines :
Electrical & electronics engineering
Author, co-author :
TRAN, Duc Dung ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SigCom
Sharma, Shree Krishna ; Interdisciplinary Center for Security,Reliability and Trust, University of Luxembourg, Luxembourg City, Luxembourg
HA, Vu Nguyen ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SigCom
CHATZINOTAS, Symeon ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SigCom
Woungang, Isaac ; Department of Computer Science, Toronto Metropolitan University, Toronto, Canada
External co-authors :
yes
Language :
English
Title :
Multi-Agent DRL Approach for Energy-Efficient Resource Allocation in URLLC-Enabled Grant-Free NOMA Systems
Publication date :
03 July 2023
Journal title :
IEEE Open Journal of the Communications Society
eISSN :
2644-125X
Publisher :
Institute of Electrical and Electronics Engineers Inc.
H. Ji, S. Park, J. Yeo, Y. Kim, J. Lee, and B. Shim, "Ultra-reliable and low-latency communications in 5G downlink: Physical layer aspects," IEEE Wireless Commun., vol. 25, no. 3, pp. 124-130, Jun. 2018.
P. Popovski et al., "Wireless access for ultra-reliable low-latency communication: Principles and building blocks," IEEE Netw., vol. 32, no. 2, pp. 16-23, Mar./Apr. 2018.
G. Durisi, T. Koch, and P. Popovski, "Toward massive, ultra-reliable, and low-latency wireless communication with short packets," Proc. IEEE, vol. 104, no. 9, pp. 1711-1726, Sep. 2016.
T. T. Nguyen, V. N. Ha, and L. B. Le, "Wireless scheduling for heterogeneous services with mixed numerology in 5G wireless networks," IEEE Commun. Lett., vol. 24, no. 2, pp. 410-413, Feb. 2020.
Y. Polyanskiy, H. V. Poor, and S. Verdu, "Channel coding rate in the finite blocklength regime," IEEE Trans. Inf. Theory, vol. 56, no. 5, pp. 2307-2359, May 2010.
S. K. Sharma and X. Wang, "Toward massive machine type communications in ultra-dense cellular IoT networks: Current issues and machine learning-Assisted solutions," IEEE Commun. Surveys Tuts., vol. 22, no. 1, pp. 426-471, 1st Quart., 2020.
A. C. Cirik, N. M. Balasubramanya, L. Lampe, G. Vos, and S. Bennett, "Toward the standardization of grant-free operation and the associated NOMA strategies in 3GPP," IEEE Commun. Stand. Mag., vol. 3, no. 4, pp. 60-66, Dec. 2019.
R. Abbas, M. Shirvanimoghaddam, Y. Li, and B. Vucetic, "A novel analytical framework for massive grant-free NOMA," IEEE Trans. Commun., vol. 67, no. 3, pp. 2436-2449, Nov. 2018.
C. Xiao et al., "Downlink MIMO-NOMA for ultra-reliable lowlatency communications," IEEE J. Sel. Areas Commun., vol. 37, no. 4, pp. 780-794, Apr. 2019.
Z. Wang, T. Lv, Z. Lin, J. Zeng, and P. T. Mathiopoulos, "Outage performance of URLLC NOMA systems with wireless power transfer," IEEE Wireless Commun. Lett., vol. 9, no. 3, pp. 380-384, Mar. 2020.
D.-D. Tran, S. K. Sharma, S. Chatzinotas, I. Woungang, and B. Ottersten, "Short-packet communications for MIMO NOMA systems over Nakagami-m fading: BLER and minimum blocklength analysis," IEEE Trans. Veh. Technol., vol. 70, no. 4, pp. 3583-3598, Apr. 2021.
5G NR, Physical Layer Procedures for Data, V15.9.0, 3GPP Standard TS 38.214, Mar. 2020.
C. Jiang, H. Zhang, Y. Ren, Z. Han, K.-C. Chen, and L. Hanzo, "Machine learning paradigms for next-generation wireless networks," IEEE Wireless Commun., vol. 24, no. 2, pp. 98-105, Apr. 2017.
H. Ye, G. Y. Li, and B. Juang, "Power of deep learning for channel estimation and signal detection in OFDM systems," IEEE Wireless Commun. Lett., vol. 7, no. 1, pp. 114-117, Feb. 2018.
H. Huang, W. Xia, J. Xiong, J. Yang, G. Zheng, and X. Zhu, "Unsupervised learning-based fast beamforming design for downlink MIMO," IEEE Access, vol. 7, pp. 7599-7605, 2019.
N. C. Luong et al., "Applications of deep reinforcement learning in communications and networking: A survey," IEEE Commun. Surveys Tuts., vol. 21, no. 4, pp. 3133-3174, 4th Quart., 2019.
J. Yu and L. Chen, "Stability analysis of frame slotted aloha protocol," IEEE Trans. Mobile Comput., vol. 16, no. 5, pp. 1462-1474, Jul. 2016.
H. Cao and J. Cai, "Distributed opportunistic spectrum access in an unknown and dynamic environment: A stochastic learning approach," IEEE Trans. Veh. Technol., vol. 67, no. 5, pp. 4454-4465, Jan. 2018.
M. Shirvanimoghaddam, M. Condoluci, M. Dohler, and S. J. Johnson, "On the fundamental limits of random non-orthogonal multiple access in cellular massive IoT," IEEE J. Sel. Areas Commun., vol. 35, no. 10, pp. 2238-2252, Oct. 2017.
S. K. Sharma and X. Wang, "Collaborative distributed Q-learning for RACH congestion minimization in cellular IoT networks," IEEE Commun. Lett., vol. 23, no. 4, pp. 600-603, Apr. 2019.
M. V. da Silva, R. D. Souza, H. Alves, and T. Abrão, "A NOMA-based Q-learning random access method for machine type communications," IEEE Wireless Commun. Lett., vol. 9, no. 10, pp. 1720-1724, Oct. 2020.
D.-D. Tran, S. K. Sharma, and S. Chatzinotas, "BLER-based adaptive Q-learning for efficient random access in NOMA-based mMTC networks," in Proc. IEEE Veh. Technol. Conf. (VTC), Helsinki, Finland, Apr. 2021, pp. 1-5.
D.-D. Tran, S. K. Sharma, S. Chatzinotas, and I. Woungang, "Learning-based multiplexing of grant-based and grant-free heterogeneous services with short packets," in Proc. IEEE Global Commun. Conf. (GLOBECOM), Madrid, Spain, Dec. 2021, pp. 1-6.
D.-D. Tran, V. N. Ha, and S. Chatzinotas, "Novel reinforcement learning based power control and subchannel selection mechanism for grant-free NOMA URLLC-enabled systems," in Proc. IEEE Veh. Technol. Conf. (VTC), 2022, pp. 1-5.
Y. Hua, R. Li, Z. Zhao, X. Chen, and H. Zhang, "GAN-powered deep distributional reinforcement learning for resource management in network slicing," IEEE J. Sel. Areas Commun., vol. 38, no. 2, pp. 334-349, Feb. 2020.
A. T. Z. Kasgari, W. Saad, M. Mozaffari, and H. V. Poor, "Experienced deep reinforcement learning with generative adversarial networks (GANs) for model-free ultra reliable low latency communication," IEEE Trans. Commun., vol. 69, no. 2, pp. 884-899, Feb. 2021.
R. Huang, V. W. S. Wong, and R. Schober, "Throughput optimization for grant-free multiple access with multiagent deep reinforcement learning," IEEE Trans. Wireless Commun., vol. 20, no. 1, pp. 228-242, Jan. 2021.
J. Zhang, X. Tao, H. Wu, N. Zhang, and X. Zhang, "Deep reinforcement learning for throughput improvement of the uplink grant-free NOMA system," IEEE Internet Things J., vol. 7, no. 7, pp. 6369-6379, Jul. 2020.
M. Fayaz, W. Yi, Y. Liu, and A. Nallanathan, "Transmit power pool design for grant-free NOMA-IoT networks via deep reinforcement learning," IEEE Trans. Wireless Commun., vol. 20, no. 11, pp. 7626-7641, Nov. 2021.
Y. Liu, Y. Deng, M. Elkashlan, and A. Nallanathan, "Cooperative deep reinforcement learning based grant-free NOMA optimization for mURLLC," in Proc. IEEE Int. Conf. Commun. (ICC), 2022, pp. 1-6.
Y. Liu, Y. Deng, H. Zhou, M. Elkashlan, and A. Nallanathan, "Deep reinforcement learning-based grant-free NOMA optimization for mURLLC," IEEE Trans. Commun., vol. 71, no. 3, pp. 1475-1490, Mar. 2023.
S.-Y. Lien, S.-L. Shieh, Y. Huang, B. Su, Y.-L. Hsu, and H.-Y. Wei, "5G new radio: Waveform, frame structure, multiple access, and initial access," IEEE Commun. Mag., vol. 55, no. 6, pp. 64-71, Jun. 2017.
V. N. Ha, T. T. Nguyen, L. B. Le, and J.-F. Frigon, "Admission control and network slicing for multi-numerology 5G wireless networks," IEEE Netw. Lett., vol. 2, no. 1, pp. 5-9, Mar. 2020.
H. Liu, N. I. Miridakis, T. A. Tsiftsis, K. J. Kim, and K. S. Kwak, "Coordinated uplink transmission for cooperative NOMA systems," in Proc. IEEE Global Commun. Conf. (GLOBECOM), Abu Dhabi, UAE, Dec. 2018, pp. 1-6.
C. Sun, C. She, C. Yang, T. Q. S. Quek, Y. Li, and B. Vucetic, "Optimizing resource allocation in the short blocklength regime for ultra-reliable and low-latency communications," IEEE Trans. Wireless Commun., vol. 18, no. 1, pp. 402-415, Jan. 2019.
Z. Sheng, D. Tian, and V. C. M. Leung, "Toward an energy and resource efficient Internet of Things: A design principle combining computation, communications, and protocols," IEEE Commun. Mag., vol. 56, no. 7, pp. 89-95, Jul. 2018.
L. Liang, H. Ye, and G. Y. Li, "Spectrum sharing in vehicular networks based on multi-Agent reinforcement learning," IEEE J. Sel. Areas Commun., vol. 37, no. 10, pp. 2282-2292, Oct. 2019.
R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA, USA: MIT Press, 2018.
V. Mnih et al., "Human-level control through deep reinforcement learning," Nature, vol. 518, no. 7540, pp. 529-533, 2015.
H. V. Hasselt, A. Guez, and D. Silver, "Deep reinforcement learning with double Q-learning," 2015, arXiv:1509.06461.
M. H. H. M. L. Z. Wang, T. Schaul and N. Freitas, "Dueling network architectures for deep reinforcement learning," 2016, arXiv:1511.06581.
X. Liu, Y. Liu, Y. Chen, and L. Hanzo, "Enhancing the fueleconomy of V2I-Assisted autonomous driving: A reinforcement learning approach," IEEE Trans. Veh. Technol., vol. 69, no. 8, pp. 8329-8342, Aug. 2020.
X. Liu, Y. Liu, and Y. Chen, "Machine learning empowered trajectory and passive beamforming design in UAV-sRIS wireless networks," IEEE J. Sel. Areas Commun., vol. 39, no. 7, pp. 2042-2055, Jul. 2021.
A. Sannai, Y. Takai, and M. Cordonnier, "Universal approximations of permutation invariant/equivariant functions by deep neural networks," 2019, arXiv:1903.01939.
F. Wu, H. Zhang, J. Wu, and L. Song, "Cellular UAV-To-device communications: Trajectory design and mode selection by multi-Agent deep reinforcement learning," IEEE Trans. Commun., vol. 68, no. 7, pp. 4175-4189, Jul. 2020.
A. Nouruzi et al., "Toward a smart resource allocation policy via artificial intelligence in 6G networks: Centralized or decentralized?" 2022, arXiv:2202.09093.
M. Zeng, X. Li, G. Li, W. Hao, and O. A. Dobre, "Sum rate maximization for IRS-Assisted uplink NOMA," IEEE Commun. Lett., vol. 25, no. 1, pp. 234-238, Jan. 2021.
