Multi-agent reinforcement learning; Vehicular cloudlet; Industry 5.0
Abstract :
[en] Automotive-Industry 5.0 will use Beyond Fifth-Generation (B5G) communications to provide robust, abundant computation resources and energy-efficient data sharing among various Intelligent Transportation System (ITS) entities. Based on the vehicle communication network, the Internet of Vehicles (IoV) is created, where vehicles’ resources, including processing, storage, sensing, and communication units, can be leveraged to construct Vehicular Cloudlet (VC) to realize resource sharing. As Connected and Autonomous Vehicles (CAV) onboard computing is becoming more potent, VC resources (comprising stationary and moving vehicles’ idle resources) seems a promising solution to tackle the incessant computing requirements of vehicles. Furthermore, such spare computing resources can significantly reduce task requests’ delay and transmission costs. In order to maximize the utility of task requests in the system under the maximum time constraint, this paper proposes a Secondary Resource Allocation (SRA) mechanism based on a dual time scale. The request service process is regarded as M/M/1 queuing model and considers each task request in the same time slot as an agent. A Partially Observable Markov Decision Process (POMDP) is constructed and combined with the Multi-Agent Reinforcement Learning (MARL) algorithm known as QMix, which exploits the overall vehicle state and queue state to reach effective computing resource allocation decisions. There are two main performance metrics: the system’s total utility and task completion rate. Simulation results reveal that the task completion rate is increased by 13%. Furthermore, compared with the deep deterministic policy optimization method, our proposed algorithm can improve the overall utility value by 70% and the task completion rate by 6%.
Disciplines :
Electrical & electronics engineering
Author, co-author :
Ahmed, Manzoor
Liu, Jinshi
Mirza, Muhammad Ayzed
Khan, Wali Ullah ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SigCom
Al-wesabi, Fahd N.
External co-authors :
yes
Language :
English
Title :
MARL based resource allocation scheme leveraging vehicular cloudlet in automotive-industry 5.0
Alternative titles :
[en] MARL based resource allocation scheme leveraging vehicular cloudlet in automotive-industry 5.0
Publication date :
19 October 2022
Journal title :
Journal of King Saud University - Computer and Information Sciences
Ahmed, M., Raza, S., Mirza, M.A., Aziz, A., Khan, M.A., Khan, W.U., Li, J., Han, Z., A survey on vehicular task offloading: Classification, issues, and challenges. J. King Saud University - Computer Inform. Sci., 2022.
Dai, Y., Xu, D., Maharjan, S., Zhang, Y., Joint load balancing and offloading in vehicular edge computing and networks. IEEE Internet Things J. 6:3 (2018), 4377–4387.
Dai, Y., Zhang, K., Maharjan, S., Zhang, Y., Edge intelligence for energy-efficient computation offloading and resource allocation in 5g beyond. IEEE Transactions on Vehicular Technology, 69(10), 2020 pp. 12 175–12 186.
Deng, S., Zhao, H., Fang, W., Yin, J., Dustdar, S., Zomaya, A.Y., Edge intelligence: the confluence of edge computing and artificial intelligence. IEEE Internet Things J. 7:8 (2020), 7457–7469.
El-Sayed, H., Chaqfeh, M., Exploiting mobile edge computing for enhancing vehicular applications in smart cities. Sensors, 19(5), 2019, 1073.
H. Van Hasselt, A. Guez, and D. Silver, ”Deep reinforcement learning with double q-learning,” in Proceedings of the AAAI conference on artificial intelligence, vol. 30, no. 1, 2016.
Jameel, F., Javaid, U., Khan, W.U., Aman, M.N., Pervaiz, H., Jäntti, R., Reinforcement learning in blockchain-enabled IIoT networks: A survey of recent advances and open challenges. Sustainability, 12(12), 2020, 5161.
Jiang, Z., Zhou, S., Guo, X., Niu, Z., Task replication for deadline-constrained vehicular cloud computing: Optimal policy, performance analysis, and implications on road traffic. IEEE Internet Things J. 5:1 (2018), 93–107.
Khan, W.U., Jamshed, M.A., Lagunas, E., Chatzinotas, S., Li, X., Ottersten, B., Energy efficiency optimization for backscatter enhanced NOMA cooperative V2X communications under imperfect CSI. IEEE Trans. Intell. Transp. Syst., 2022.
Khan, W.U., Ihsan, A., Nguyen, T.N., Javed, M.A., Ali, Z., NOMA-enabled backscatter communications for green transportation in automotive-Industry 5.0. IEEE Trans. Industr. Inf., 2022.
Lee, S.-S., Lee, S., Resource allocation for vehicular fog computing using reinforcement learning combined with heuristic information. IEEE Internet Things J., 7(10), 2020 pp. 10 450–10 464.
Li, B., Fei, Z., Chu, Z., Zhang, Y., Secure transmission for heterogeneous cellular networks with wireless information and power transfer. IEEE Syst. J. 12:4 (2018), 3755–3766.
Lin, C., Deng, D., Yao, C., Resource allocation in vehicular cloud computing systems with heterogeneous vehicles and roadside units. IEEE Internet Things J. 5:5 (2018), 3692–3700.
Liu, J., Ahmed, M., Mirza, M.A., Khan, W.U., Xu, D., Li, J., Aziz, A., Han, Z., Rl/drl meets vehicular task offloading using edge and vehicular cloudlet: A survey. IEEE Internet Things J. 9:11 (2022), 8315–8338.
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., Mordatch, I., 2017. Multi-agent actor-critic for mixed cooperative-competitive environments, vol. 30.
Mahmood, A., Hong, Y., Ehsan, M.K., Mumtaz, S., Optimal resource allocation and task segmentation in IoT enabled mobile edge cloud. IEEE Transactions on Vehicular Technology, 70(12), 2021 pp. 13 294–13 303.
Malik, U.M., Javed, M.A., Zeadally, S., Islam, S., Energy efficient fog computing for 6g enabled massive iot: Recent trends and future opportunities. IEEE Internet Things J., 2021, 1.
Ning, Z., Dong, P., Wang, X., Guo, L., Rodrigues, J.J., Kong, X., Huang, J., Kwok, R.Y., Deep reinforcement learning for intelligent internet of vehicles: An energy-efficient computational offloading scheme. IEEE Transactions on Cognitive Communications and Networking 5:4 (2019), 1060–1072.
Pu, C., Carpenter, L., Psched: A priority-based service scheduling scheme for the internet of drones. IEEE Syst. J. 15 (2021), 4230–4239.
Qi, Q., Wang, J., Ma, Z., Sun, H., Cao, Y., Zhang, L., Liao, J., Knowledge-driven service offloading decision for vehicular edge computing: A deep reinforcement learning approach. IEEE Trans. Veh. Technol. 68:5 (2019), 4192–4203.
Qu, Y., Pokhrel, S.R., Garg, S., Gao, L., Xiang, Y., A blockchained federated learning framework for cognitive computing in industry 4.0 networks. IEEE Trans. Industr. Inf. 17 (2021), 2964–2973.
Rashid, T., Samvelyan, M., Witt, C.S.D., Farquhar, G., Foerster, J.N., Whiteson, S., Monotonic value function factorisation for deep multi-agent reinforcement learning. J. Mach. Learn. Res., 21, 2020 pp. 178:1–178:51.
Raza, S., Wang, S., Ahmed, M., Anwar, M.R., A survey on vehicular edge computing: Architecture, applications, technical issues, and future directions. Wireless Communications and Mobile Computing, 2019, 2019.
Raza, S., Liu, W., Ahmed, M., Anwar, M.R., Mirza, M.A., Sun, Q., Wang, S., An efficient task offloading scheme in vehicular edge computing. J. Cloud Computing 9:1 (2020), 1–14.
Sheraz, M., Ahmed, M., Hou, X., Li, Y., Jin, D., Han, Z., Jiang, T., Artificial intelligence for wireless caching: Schemes, performance, and challenges. IEEE Commun. Surveys Tutorials 23:1 (2021), 631–661.
Skondras, E., Michalas, A., Vergados, D.D., Mobility management on 5g vehicular cloud computing systems. Vehicular Communications 16 (2019), 15–44.
Spinelli, F., Mancuso, V., Towards enabled industrial verticals in 5g: a survey on mec-based approaches to provisioning and flexibility. IEEE Communications Surveys & Tutorials, 2020.
Sun, F., Cheng, N., Zhang, S., Zhou, H., Gui, L., Shen, X., 2018. “Reinforcement learning based computation migration for vehicular cloud computing”. In: 2018 IEEE Global Communications Conference (GLOBECOM), Dec. 2018, pp. 1–6.
Tokody, D., Albini, A., Ady, L., Rajnai, Z., Pongrácz, F., Safety and security through the design of autonomous intelligent vehicle systems and intelligent infrastructure in the smart city. Interdisciplinary Description of Complex Systems: INDECS 3-A (2018), 384–396.
Ullah, S., Abbas, G., Abbas, Z.H., Waqas, M., Ahmed, M., Rbo-em: Reduced broadcast overhead scheme for emergency message dissemination in vanets. IEEE Access, 8, 2020 pp. 175 205–175 219.
Wang, Z., Zhong, Z., Ni, M., 2018. “Application-aware offloading policy using smdp in vehicular fog computing systems”. In: 2018 IEEE International Conference on Communications Workshops (ICC Workshops), May. 2018, pp. 1–6.
Xiao, Z., Dai, X., Jiang, H., Wang, D., Chen, H., Yang, L., Zeng, F., Vehicular task offloading via heat-aware mec cooperation using game-theoretic method. IEEE Internet Things J. 7:3 (2019), 2038–2052.
Zhang, Y., Lopez, J., Wang, Z., Mobile edge computing for vehicular networks [from the guest editors]. IEEE Veh. Technol. Mag. 14:1 (2019), 27–108.
Zhang, K., Leng, S., Peng, X., Pan, L., Maharjan, S., Zhang, Y., Artificial intelligence inspired transmission scheduling in cognitive vehicular communications and networks. IEEE Internet Things J. 6:2 (2019), 1987–1997.
Zhao, J., Li, Q., Gong, Y., Zhang, K., Computation offloading and resource allocation for cloud assisted mobile edge computing in vehicular networks. IEEE Trans. Veh. Technol. 68:8 (2019), 7944–7956.
Zhou, Z., Liu, P., Feng, J., Zhang, Y., Mumtaz, S., Rodriguez, J., Computation resource allocation and task assignment optimization in vehicular fog computing: A contract-matching approach. IEEE Trans. Veh. Technol. 68:4 (2019), 3113–3125.
