[en] Maintenance planning and scheduling are an essential part of manufacturing companies to prevent machine breakdowns and increase machine uptime, along with production efficiency. One of the biggest challenges is to effectively address uncertainty (e.g., unexpected machine failures, variable time to repair). Multiple approaches have been used to solve the maintenance scheduling problem, including dispatching rules (DR), metaheuristics and simheuristics, or most recently reinforcement learning (RL). However, to the best of our knowledge, no study has ever studied to what extent these techniques are effective when faced with different levels of uncertainty. To overcome this gap in research, this paper presents an approach by analyzing the impact of categorized levels of uncertainty, specifically high and low, on the failure distribution and time to repair. Upon the formalization of the maintenance scheduling problem, the experiments conducted are performed in simulated scenarios with different degrees of uncertainty, and also considering a real-life manufacturing use case. The results indicate that rescheduling based on a genetic algorithm (GA) simheuristic outperforms RL and DR in terms of total machine uptime, but not in terms of the mean time to repair when configured with high re-optimization frequencies (i.e., hourly re-optimization), but rapidly underperforms when the re-optimization frequency decreases. Furthermore, our study demonstrates that GA-simheuristic is highly computationally demanding compared to RL and rule-based policies.
Disciplines :
Computer science
Author, co-author :
RUIZ RODRIGUEZ, Marcelo Luis ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SerVal
KUBLER, Sylvain ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SerVal
Robert, Jérémy; Cebi Luxembourg S.A.
LE TRAON, Yves ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT)
External co-authors :
no
Language :
English
Title :
Dynamic maintenance scheduling approach under uncertainty: Comparison between reinforcement learning, genetic algorithm simheuristic, dispatching rules
This research was funded in whole or in part by the Luxembourg National Research Fund (FNR) , grant reference 16756339 . For the purpose of open access, and in fulfillment of the obligations arising from the grant agreement, the author has applied a Creative Commons Attribution 4.0 International (CC BY 4.0) license to any Author Accepted Manuscript version arising from this submission.
Alves, F.F., Ravetti, M.G., Hybrid proactive approach for solving maintenance and planning problems in the scenario of Industry 4.0. IFAC-PapersOnLine 53:3 (2020), 216–221, 10.1016/j.ifacol.2020.11.035.
Arena, M., Di Pasquale, V., Iannone, R., Miranda, S., Riemma, S., A maintenance driven scheduling cockpit for integrated production and maintenance operation schedule. Advances in Manufacturing 10:2 (2022), 205–219, 10.1007/s40436-021-00380-z.
Benaggoune, K., Meraghni, S., Ma, J., Mouss, L.H., Zerhouni, N., Post prognostic decision for predictive maintenance planning with remaining useful life uncertainty. Proceedings - 2020 prognostics and health management conference, 2020, 194–199, 10.1109/PHM-Besancon49106.2020.00039.
Bencheikh, G., Letouzey, A., Desforges, X., An approach for joint scheduling of production and predictive maintenance activities. Journal of Manufacturing Systems 64 (2022), 546–560, 10.1016/j.jmsy.2022.08.005.
Celen, M., Djurdjanovic, D., Integrated maintenance and operations decision making with imperfect degradation state observations. Journal of Manufacturing Systems 55 (2020), 302–316, 10.1016/j.jmsy.2020.03.010.
De Jonge, B., Klingenberg, W., Teunter, R., Tinga, T., Optimum maintenance strategy under uncertainty in the lifetime distribution. Reliability Engineering & System Safety 133 (2015), 59–67, 10.1016/j.ress.2014.09.013.
De Jonge, B., Scarf, P.A., A review on maintenance optimization. European Journal of Operational Research 285:3 (2020), 805–824, 10.1016/j.ejor.2019.09.047.
Deng, S., Zhu, Y., Yu, Y., Huang, X., An integrated approach of ensemble learning methods for stock index prediction using investor sentiments. Expert Systems with Applications, 238, 2024, 121710, 10.1016/j.eswa.2023.121710.
Detti, P., Nicosia, G., Pacifici, A., Zabalo Manrique de Lara, G., Robust single machine scheduling with a flexible maintenance activity. Computers & Operations Research 107 (2019), 19–31, 10.1016/j.cor.2019.03.001.
Ferjani, A., Ammar, A., Pierreval, H., Elkosantini, S., A simulation–optimization based heuristic for the online assignment of multi-skilled workers subjected to fatigue in manufacturing systems. Computers & Industrial Engineering 112 (2017), 663–674, 10.1016/j.cie.2017.02.008.
Geurtsen, M., Didden, J.B.H.C., Adan, J., Atan, Z., Adan, I., Production, maintenance and resource scheduling: A review. European Journal of Operational Research 305:2 (2023), 501–529, 10.1016/j.ejor.2022.03.045.
Ghaleb, M., Taghipour, S., Sharifi, M., Zolfagharinia, H., Integrated production and maintenance scheduling for a single degrading machine with deterioration-based failures. Computers & Industrial Engineering, 143, 2020, 106432, 10.1016/j.cie.2020.106432.
Guan, S., Zhuang, Z., Tao, H., Chen, Y., Stojanovic, V., Paszke, W., Feedback-aided PD-type iterative learning control for time-varying systems with non-uniform trial lengths. Transactions of the Institute of Measurement and Control 45:11 (2023), 2015–2026, 10.1177/01423312221142564.
Huang, J., Chang, Q., Arinez, J., Deep reinforcement learning based preventive maintenance policy for serial production lines. Expert Systems with Applications, 160, 2020, 113701, 10.1016/j.eswa.2020.113701.
Kaufmann, E., Bauersfeld, L., Loquercio, A., Müller, M., Koltun, V., Scaramuzza, D., Champion-level drone racing using deep reinforcement learning. Nature 620:7976 (2023), 982–987, 10.1038/s41586-023-06419-4.
Kuhnle, A., Jakubik, J., Lanza, G., Reinforcement learning for opportunistic maintenance optimization. Production Engineering 13:1 (2019), 33–41, 10.1007/s11740-018-0855-7.
Lu, B., Chen, Z., Zhao, X., Data-driven dynamic predictive maintenance for a manufacturing system with quality deterioration and online sensors. Reliability Engineering & System Safety, 212, 2021, 107628, 10.1016/j.ress.2021.107628.
Lutz, I.D., Wang, S., Norn, C., Courbet, A., Borst, A.J., Zhao, Y.T., Dosey, A., Cao, L., Xu, J., Leaf, E.M., Treichel, C., Litvicov, P., Li, P., Goodson, A.D., Rivera-Sánchez, P., Bratovianu, A.M., Baek, M., King, N.P., Ruohola-Baker, H., Baker, D., Top-down design of protein architectures with reinforcement learning. Science 380:6642 (2023), 266–273, 10.1126/science.adf6591.
Mi, S., Feng, Y., Zheng, H., Li, Z., Gao, Y., Tan, J., Integrated intelligent green scheduling of predictive maintenance for complex equipment based on information services. IEEE Access 8 (2020), 45797–45812, 10.1109/ACCESS.2020.2977667.
Miao, B., Deng, Q., Zhang, L., Huo, Z., Liu, X., Collaborative scheduling of spare parts production and service workers driven by distributed maintenance demand. Journal of Manufacturing Systems 64 (2022), 261–274, 10.1016/j.jmsy.2022.06.012.
Nasruddin, Nasution, S., Aisyah, N., Surachman, A., Wibowo, A.S., Exergy analysis and exergoeconomic optimization of a binary cycle system using a multi objective genetic algorithm. International Journal of Technology 9:2 (2018), 275–286, 10.14716/ijtech.v9i2.1040.
Németh, I., Kocsis, Á., Takács, D., Shaheen, B.W., Takács, M., Merlo, A., Eytan, A., Bidoggia, L., Olocco, P., Maintenance schedule optimisation for manufacturing systems. IFAC-PapersOnLine 53:3 (2020), 319–324, 10.1016/j.ifacol.2020.11.051.
Neumann, A., Gounder, S., Yan, X., Sherman, G., Campbell, B., Guo, M., Neumann, F., Diversity optimization for the detection and concealment of spatially defined communication networks. GECCO 2023 - proceedings of the 2023 genetic and evolutionary computation conference, 2023, 1436–1444, 10.1145/3583131.3590405.
Pan, W., Liu, S.Q., Deep reinforcement learning for the dynamic and uncertain vehicle routing problem. Applied Intelligence 53:1 (2023), 405–422, 10.1007/s10489-022-03456-w.
Rokhforoz, P., Fink, O., Distributed joint dynamic maintenance and production scheduling in manufacturing systems: Framework based on model predictive control and benders decomposition. Journal of Manufacturing Systems 59 (2021), 596–606, 10.1016/j.jmsy.2021.04.010.
Ruiz Rodríguez, M.L., Kubler, S., de Giorgio, A., Cordy, M., Robert, J., Le Traon, Y., Multi-agent deep reinforcement learning based predictive maintenance on parallel machines. Robotics and Computer-Integrated Manufacturing, 78, 2022, 102406, 10.1016/j.rcim.2022.102406.
Ruschel, E., Santos, E.A.P., Loures, E. de F.R., Establishment of maintenance inspection intervals: an application of process mining techniques in manufacturing. Journal of Intelligent Manufacturing 31:1 (2020), 53–72, 10.1007/s10845-018-1434-7.
Sarazin, A., Bascans, J., Sciau, J.B., Song, J., Supiot, B., Montarnal, A., Lorca, X., Truptil, S., Expert system dedicated to condition-based maintenance based on a knowledge graph approach: Application to an aeronautic system. Expert Systems with Applications, 186, 2021, 115767, 10.1016/j.eswa.2021.115767.
Song, X., Sun, P., Song, S., Stojanovic, V., Finite-time adaptive neural resilient DSC for fractional-order nonlinear large-scale systems against sensor-actuator faults. Nonlinear Dynamics 111:13 (2023), 12181–12196, 10.1007/s11071-023-08456-0.
Stojanovic, V., Fault-tolerant control of a hydraulic servo actuator via adaptive dynamic programming. Mathematical Modelling and Control 3:3 (2023), 181–191, 10.3934/mmc.2023016.
Su, J., Huang, J., Adams, S., Chang, Q., Beling, P.A., Deep multi-agent reinforcement learning for multi-level preventive maintenance in manufacturing systems. Expert Systems with Applications, 192, 2022, 116323, 10.1016/j.eswa.2021.116323.
Sulewski, P., Szymkowiak, M., The Weibull lifetime model with randomised failure-free time. Statistics in Transition New Series 23:4 (2022), 59–76, 10.2478/stattrans-2022-0042.
Sun, Q., Ye, Z.S., Peng, W., Scheduling preventive maintenance considering the saturation effect. IEEE Transactions on Reliability 68:2 (2019), 741–752, 10.1109/TR.2018.2874265.
Valet, A., Altenmüller, T., Waschneck, B., May, M.C., Kuhnle, A., Lanza, G., Opportunistic maintenance scheduling with deep reinforcement learning. Journal of Manufacturing Systems 64 (2022), 518–534, 10.1016/j.jmsy.2022.07.016.
Wocker, M., Betz, N.K., Feuersänger, C., Lindworsky, A., Deuse, J., Unsupervised learning for opportunistic maintenance optimization in flexible manufacturing systems. Procedia CIRP 93 (2020), 1025–1030, 10.1016/j.procir.2020.04.025.
Xia, T., Shi, G., Si, G., Du, S., Xi, L., Energy-oriented joint optimization of machine maintenance and tool replacement in sustainable manufacturing. Journal of Manufacturing Systems 59 (2021), 261–271, 10.1016/j.jmsy.2021.01.015.
Yan, Q., Wang, H., Wu, F., Digital twin-enabled dynamic scheduling with preventive maintenance using a double-layer Q-learning algorithm. Computers & Operations Research, 144, 2022, 105823, 10.1016/j.cor.2022.105823.
Yang, C.Y., Shiranthika, C., Wang, C.Y., Chen, K.W., Sumathipala, S., Reinforcement learning strategies in cancer chemotherapy treatments: A review. Computer Methods and Programs in Biomedicine, 229, 2023, 107280, 10.1016/j.cmpb.2022.107280.
Yazdani, R., Alipour-Vaezi, M., Kabirifar, K., Salahi Kojour, A., Soleimani, F., A lion optimization algorithm for an integrating maintenance planning and production scheduling problem with a total absolute deviation of completion times objective. Soft Computing 26:24 (2022), 13953–13968, 10.1007/s00500-022-07436-7.
Ying, K.C., Lu, C.C., Chen, J.C., Exact algorithms for single-machine scheduling problems with a variable maintenance. Computers & Industrial Engineering 98 (2016), 427–433, 10.1016/j.cie.2016.05.037.
Yu, T., Zhu, C., Chang, Q., Wang, J., Imperfect corrective maintenance scheduling for energy efficient manufacturing systems through online task allocation method. Journal of Manufacturing Systems 53 (2019), 282–290, 10.1016/j.jmsy.2019.11.002.
Zahir, A.A.M., Alhady, S.S.N., Othman, W.A.F.W., Wahab, A.A.A., Ahmad, M.F., Objective functions modification of GA optimized PID controller for brushed DC motor. International Journal of Electrical and Computer Engineering 10:3 (2020), 2426–2433, 10.11591/ijece.v10i3.pp2426-2433.
Zhuang, Z., Tao, H., Chen, Y., Stojanovic, V., Paszke, W., An optimal iterative learning control approach for linear systems with nonuniform trial lengths under input constraints. IEEE Transactions on Systems, Man, and Cybernetics: Systems 53:6 (2023), 3461–3473, 10.1109/TSMC.2022.3225381.
