Evaluating the effectiveness of LLMs for explainable deep reinforcement learning

BELOUADAH, Ayoub; RUIZ RODRIGUEZ, Marcelo Luis; KUBLER, Sylvain; Le Traon, Yves

doi:10.1016/j.mlwa.2025.100795

Download

Article (Scientific journals)

Evaluating the effectiveness of LLMs for explainable deep reinforcement learning

BELOUADAH, Ayoub; RUIZ RODRIGUEZ, Marcelo Luis; KUBLER, Sylvain et al.

2025 • In Machine Learning with Applications, 22, p. 100795

Peer reviewed

Permalink
https://hdl.handle.net/10993/66628

DOI
10.1016/j.mlwa.2025.100795

Files (1)Send to Details Statistics Bibliography Similar publications

Files

Full Text

1-s2.0-S2666827025001781-main.pdf

Author postprint (2.09 MB)

Download

All documents in ORBilu are protected by a user license.

Send to

RIS BibTex APA Chicago Permalink X Linkedin

Details

Keywords :

Large language models; Reinforcement learning; Explainable Reinforcement learning; Monte Carlo Tree Search; LLM as a judge

Abstract :

[en] Understanding the decision-making of reinforcement learning (RL) agents is essential for real-world deployment. Existing eXplainable RL (XRL) techniques, such as feature attribution and policy visualization, provide insight but remain inaccessible to non-experts. Large Language Models (LLMs) offer a natural-language alternative, yet often lack logical consistency and alignment with agent goals. This study benchmarks three explanation generation methods: Chain-of-Thought (CoT) prompting as the standard baseline used in prior work, Monte Carlo Tree Search (MCTS) augmentation, and supervised fine-tuning (SFT) across various models. Evaluations using Soundness and Fidelity show that CoT frequently produces reasoning errors, whereas MCTS improves quality for larger models (avg. +23% Soundness, +17% Fidelity), while SFT yields greater and more consistent gains for smaller ones (+58% Soundness, +52% Fidelity), underscoring the need to align methods with model capacity. An LLM-as-a-Judge framework further validates these findings, showing strong agreement with human assessments (weighted Cohen’s 𝜅 = 0.77, Spearman 𝜌 = 0.88), supporting scalable and reliable assessment of textual explanations.

Research center :

Interdisciplinary Centre for Security, Reliability and Trust (SnT) > SerVal - Security, Reasoning & Validation

Disciplines :

Computer science

Author, co-author :

BELOUADAH, Ayoub ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SerVal

RUIZ RODRIGUEZ, Marcelo Luis ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SerVal

KUBLER, Sylvain ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SerVal

Le Traon, Yves

External co-authors :

Language :

English

Title :

Evaluating the effectiveness of LLMs for explainable deep reinforcement learning

Publication date :

14 November 2025

Journal title :

Machine Learning with Applications

ISSN :

2666-8270

Publisher :

Elsevier BV

Volume :

Pages :

100795

Peer reviewed :

Peer reviewed

Additional URL :

https://api.elsevier.com/content/article/PII:S2666827025001781?httpAccept=text/xml

FnR Project :

FNR16756339 - UPTIME4.0 - Robust Predictive Maintenance For Industry 4.0, 2021 (01/06/2022-31/05/2025) - Yves Le Traon

Funders :

National Research Fund

Funding number :

16756339

Funding text :

This research was funded in whole or in part by the Luxembourg National Research Fund (FNR) , grant reference 16756339. For the purpose of open access, the author has applied a Creative Commons Attribution 4.0 International (CC BY 4.0) license to any Author Accepted Manuscript version arising from this submission.

Available on ORBilu :

since 04 December 2025

Statistics

Number of views

42 (11 by Unilu)

Number of downloads

29 (3 by Unilu)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

OpenAlex citations

Bibliography

Abu-Rasheed, H., Weber, C., & Fathi, M. (2024). Knowledge graphs as context sources for llm-based explanations of learning recommendations. In 2024 IEEE Global Engineering Education Conference (EDUCON) (p. 1–5). IEEE.
Ahmad, W., Shahzad, A. R., Amin, M. A., Bangyal, W. H., Alahmadi, T. J., & Khan, S. H. (2025). Machine learning driven dashboard for chronic myeloid leukemia prediction using protein sequences. PLoS One, 20, e0321761.
Ali, A., Xia, Y., Zia, M. F., Bangyal, W. H., & Iqbal, M. (2025). Trustworthy load forecasting with generative ai: A dual-attention convlstm and vae-based approach. IEEE Transactions on Consumer Electronics, .
Ameur, M., Brik, B., & Ksentini, A. (2024). Leveraging llms to explain drl decisions for transparent 6g network slicing. In 2024 IEEE 10th International Conference on Network Softwarization (NetSoft) (pp. 204–212). IEEE.
Ashraf, A., Qingjie, Z., Bangyal, W. H. K., & Iqbal, M. (2023). Analysis of brain imaging data for the detection of early age autism spectrum disorder using transfer learning approaches for internet of things. IEEE Transactions on Consumer Electronics, 70, 4478–4489.
Ashwood, Z., Jha, A., & Pillow, J. W. (2022). Dynamic inverse reinforcement learning for characterizing animal behavior. Advances in neural information processing systems, 35, 29663–29676.
Atrey, A., Clary, K., & Jensen, D. (2020). Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep RL. In International conference on learning representations.
Attai, K., Ekpenyong, M., Amannah, C., Asuquo, D., Ajuga, P., Obot, O., Johnson, E., John, A., Maduka, O., Akwaowo, C., & Uzoka, F.-M. (2024). Enhancing the interpretability of malaria and typhoid diagnosis with explainable ai and large language models. Tropical Medicine and Infectious Disease, 9, 216.
Badshah, S., & Sajjad, H. (2024). Reference-guided verdict: Llms-as-judges in automatic evaluation of free-form text. arXiv preprint arXiv:2408.09235, .
Bastani, O., Pu, Y., & Solar-Lezama, A. (2018). Verifiable reinforcement learning via policy extraction. In Proceedings of the 32nd International Conference on Neural Information Processing Systems NIPS’18 (p. 2499–2509). Red Hook, NY, USA: Curran Associates Inc.
Beechey, D., Smith, T. M. S., & Simşek, O.̧ (2023). Explaining reinforcement learning with shapley values. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, & J. Scarlett (Eds.), Proceedings of the 40th International Conference on Machine Learning (pp. 2003–2014). PMLR volume 202 of Proceedings of Machine Learning Research.
Boggess, K., Kraus, S., & Feng, L. (2022). Toward policy explanations for multi-agent reinforcement learning. arXiv preprint arXiv:2204.12568, .
de Souza Loureiro, A., Valverde-Rebaza, J., Noguez, J., Escarcega, D., & Marcacini, R. (2025). Advancing multi-step mathematical reasoning in large language models through multi-layered self-reflection with auto-prompting. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 205–223). Springer.
Evans, B. D., Jordaan, H. W., & Engelbrecht, H. A. (2023). Comparing deep reinforcement learning architectures for autonomous racing. Machine Learning with Applications, 14, 100496.
Feit, F., Metzger, A., & Pohl, K. (2022). Explaining online reinforcement learning decisions of self-adaptive systems. In 2022 IEEE international conference on autonomic computing and self-organizing systems (ACSOS) (pp. 51–60). IEEE.
Feldhus, N., Wang, Q., Anikina, T., Chopra, S., Oguz, C., & Möller, S. (2023). Interrolang: Exploring nlp models and datasets through dialogue-based explanations. arXiv preprint arXiv:2310.05592, .
Fukuchi, Y., Osawa, M., Yamakawa, H., & Imai, M. (2017). Autonomous self-explanation of behavior for interactive reinforcement learning agents. In Proceedings of the 5th International Conference on Human Agent Interaction HAI 201917 (p. 97–101). ACM.
Graham, B. (2025). mcts-llm: Monte carlo tree search for llms (github repository). https://github.com/BrendanGraham14/mcts-llm. Accessed: 2025-09-01.
Greydanus, S., Koul, A., Dodge, J., & Fern, A. (2018). Visualizing and understanding atari agents. In International conference on machine learning (pp. 1792–1801). PMLR.
Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., Wang, Y., & Guo, J. (2024). A survey on llm-as-a-judge. ArXiv, abs/2411.15594.
Gupta, K. D., Sadman, N., Sadmanee, A., Sarker, M. K., & George, R. (2023). Behavioral recommendation engine driven by only non-identifiable user data. Machine Learning with Applications, 11, 100442.
Hu, X., Liu, A., & Dai, Y. (2024). Combining chatgpt and knowledge graph for explainable machine learning-driven design: a case study. Journal of Engineering Design, (pp. 1–23).
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2022). Lora: Low-rank adaptation of large language models. In ICLR. OpenReview.net.
Huber, T., Schiller, D., & André, E. (2019). Enhancing explainability of deep reinforcement learning through selective layer-wise relevance propagation. In KI 2019: Advances in Artificial Intelligence: 42nd German Conference on AI, Kassel, Germany, September 23–26, 2019, Proceedings 42 (pp. 188–202). Springer.
Itaya, H., Hirakawa, T., Yamashita, T., Fujiyoshi, H., & Sugiura, K. (2021). Visual explanation using attention mechanism in actor-critic-based deep reinforcement learning. In 2021 International Joint Conference on Neural Networks (IJCNN) (pp. 1–10). IEEE.
Jiang, Y., Gu, S. S., Murphy, K. P., & Finn, C. (2019). Language as an abstraction for hierarchical deep reinforcement learning. Advances in neural information processing systems, 32.
Jiang, Z., & Luo, S. (2019). Neural logic reinforcement learning. In International conference on machine learning (pp. 3110–3119). PMLR.
Juozapaitis, Z., Koul, A., Fern, A., Erwig, M., & Doshi-Velez, F. (2019). Explainable reinforcement learning via reward decomposition. In IJCAI/ECAI workshop on explainable artificial intelligence.
Kamrani, A. S., Dini, A., Dagdougui, H., & Sheshyekani, K. (2025). Multi-agent deep reinforcement learning with online and fair optimal dispatch of ev aggregators. Machine Learning with Applications, 19, 100620.
Kim, H., Chen, H., Li, C., & Lee, J. M. (2025). Talktoagent: A human-centric explanation of reinforcement learning agents with large language models. ArXiv, abs/2509.04809.
Kroeger, N., Ley, D., Krishna, S., Agarwal, C., & Lakkaraju, H. (2023). In-context explainers: Harnessing llms for explaining black box models.
Kwon, M., Xie, S. M., Bullard, K., & Sadigh, D. (2023). Reward design with language models. arXiv preprint arXiv:2303.00001, .
Lee, Y., Kim, J., Kim, J., Cho, H., Kang, J., Kang, P., & Kim, N. (2024). Checkeval: A reliable llm-as-a-judge framework for evaluating text generation using checklists. arXiv preprint arXiv:2403.18771, .
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, .
Lu, W., Zhao, X., Spisak, J., Lee, J. H., & Wermter, S. (2025). Mental modelling of reinforcement learning agents by language models. Transactions on Machine Learning Research, .
Lyu, D., Yang, F., Liu, B., & Gustafson, S. (2019). Sdrl: interpretable and data-efficient deep reinforcement learning leveraging symbolic planning. In Proceedings of the AAAI Conference on Artificial Intelligence (pp. 2970–2977). volume 33.
Martens, D., Hinns, J., Dams, C., Vergouwen, M., & Evgeniou, T. (2025). Tell me a story! narrative-driven xai with large language models. Decision Support Systems, (p. 114402).
Masadome, S., & Harada, T. (2025). Reward design using large language models for natural language explanation of reinforcement learning agent actions. IEEJ Transactions on Electrical and Electronic Engineering, .
Mavrepis, P., Makridis, G., Fatouros, G., Koukos, V., Separdani, M. M., & Kyriazis, D. (2024). Xai for all: Can large language models simplify explainable ai? arXiv preprint arXiv:2401.13110, .
Mekrache, A., Mekki, M., Ksentini, A., Brik, B., & Verikoukis, C. (2024). On combining xai and llms for trustworthy zero-touch network and service management in 6g. IEEE Communications Magazine, .
Metzger, A., Bartel, J., & Laufer, J. (2023). An ai chatbot for explaining deep reinforcement learning decisions of service-oriented systems. In International Conference on Service-Oriented Computing (pp. 323–338). Springer.
Milani, S., Topin, N., Veloso, M., & Fang, F. (2024). Explainable reinforcement learning: A survey and comparative review. ACM Computing Surveys, 56, 1–36.
Milani, S., Zhang, Z., Topin, N., Shi, Z. R., Kamhoua, C., Papalexakis, E. E., & Fang, F. (2022). Maviper: Learning decision tree policies for interpretable multi-agent reinforcement learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 251–266). Springer.
Mirchandani, S., Karamcheti, S., & Sadigh, D. (2021). Ella: Exploration through learned language abstraction. Advances in neural information processing systems, 34, 29529–29540.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., & Hassabis, D. (2015). Human-level control through deep reinforcement learning. Nature, 518, 529–533.
Mott, A., Zoran, D., Chrzanowski, M., Wierstra, D., & Jimenez Rezende, D. (2019). Towards interpretable reinforcement learning using attention augmented agents. Advances in neural information processing systems, 32.
Nangue Tasse, G., James, S., & Rosman, B. (2020). A boolean task algebra for reinforcement learning. Advances in Neural Information Processing Systems, 33, 9497–9507.
Olson, M. L., Khanna, R., Neal, L., Li, F., & Wong, W.-K. (2021). Counterfactual state explanations for reinforcement learning agents via generative deep learning. Artificial Intelligence, 295, 103455.
OpenAI, Akkaya, I., Andrychowicz, M., Chociej, M., teusz Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., Schneider, J., Tezak, N. A., Tworek, J., Welinder, P., Weng, L., Yuan, Q., Zaremba, W., & Zhang, L. M. (2019). Solving rubik’s cube with a robot hand. ArXiv, abs/1910.07113.
Pandya, R., Zhao, M., Liu, C., Simmons, R., & Admoni, H. (2024). Multi-agent strategy explanations for human–robot collaboration. In 2024 IEEE International Conference on Robotics and Automation (ICRA) (pp. 17351–17357). IEEE.
Payani, A., & Fekri, F. (2020). Incorporating relational background knowledge into reinforcement learning via differentiable inductive logic programming. arXiv preprint arXiv:2003.10386, .
Peng, X., Riedl, M., & Ammanabrolu, P. (2022). Inherently explainable reinforcement learning in natural language. Advances in Neural Information Processing Systems, 35, 16178–16190.
Qing, Y., Liu, S., Song, J., Wang, H., & Song, M. (2022). A survey on explainable reinforcement learning: Concepts, algorithms, challenges. arXiv preprint arXiv:2211.06665, .
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., & Dormann, N. (2021). Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22, 1–8.
Rezazadeh, F., Chergui, H., & Mangues-Bafalluy, J. (2023). Explanation-guided deep reinforcement learning for trustworthy 6g ran slicing. In 2023 ieee international conference on communications workshops (icc workshops) (pp. 1026–1031). IEEE.
Rodriguez, I. D. J., Killian, T. W., Son, S., & Gombolay, M. C. (2020). Optimization methods for interpretable differentiable decision trees in reinforcement learning. In Proc. of the 23rd int. conf. on artificial intelligence and statistics.
Ruiz-Rodríguez, M. L., Kubler, S., Robert, J., Voisin, A., & Le Traon, Y. (2025). Evolutionary multi-objective multi-agent deep reinforcement learning for sustainable maintenance scheduling. Engineering Applications of Artificial Intelligence, 156, 111126.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, .
Shu, T., Xiong, C., & Socher, R. (2017). Hierarchical and interpretable skill acquisition in multi-task reinforcement learning. arXiv preprint arXiv:1712.07294, .
Slack, D., Krishna, S., Lakkaraju, H., & Singh, S. (2023). Explaining machine learning models with interactive natural language conversations using TalkToModel. In Nature Machine Intelligence (pp. 873–883). volume 5.
Spitzer, P., Celis, S., Martin, D., Kühl, N., & Satzger, G. (2024). Looking Through the Deep Glasses: How Large Language Models Enhance Explainability of Deep Learning Models. In Proceedings of mensch und computer 2024 (pp. 566–570).
Stein, G. J. (2021). Generating high-quality explanations for navigation in partially-revealed environments. In A. Beygelzimer, Y. Dauphin, P. Liang, & J. W. Vaughan (Eds.), Advances in Neural Information Processing Systems.
Sundararajan, M., Taly, A., & Yan, Q. (2017). Axiomatic attribution for deep networks. In International conference on machine learning (pp. 3319–3328). PMLR.
Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction. (2nd ed.). The MIT Press.
Tan, H., Yan, H., & Yang, Y. (2025). Llm-guided reinforcement learning: Addressing training bottlenecks through policy modulation. ArXiv, abs/2505.20671.
Topin, N., Milani, S., Fang, F., & Veloso, M. (2021). Iterative bounding mdps: Learning interpretable policies via non-interpretable methods. In Proceedings of the AAAI Conference on Artificial Intelligence (pp. 9923–9931). volume 35.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., & Lample, G. (2023). Llama: Open and efficient foundation language models.
Towers, M., Du, Y., Freeman, C., & Norman, T. J. (2024). Explaining an agent’s future beliefs through temporally decomposing future reward estimators. arXiv preprint arXiv:2408.08230, .
Wang, J., Liang, Y., Meng, F., Sun, Z., Shi, H., Li, Z., Xu, J., Qu, J., & Zhou, J. (2023). Is ChatGPT a good NLG evaluator? a preliminary study. In Y. Dong, W. Xiao, L. Wang, F. Liu, & G. Carenini (Eds.), Proceedings of the 4th New Frontiers in Summarization Workshop (pp. 1–11). Singapore: Association for Computational Linguistics.
Yildirim, M., Dagda, B., Asodia, V., & Fallah, S. (2025). Highwayllm: Decision-making and navigation in highway driving with rl-informed language model. Robotics and Autonomous Systems, (p. 105114).
Zhang, X., Guo, Y., Stepputtis, S., Sycara, K., & Campbell, J. (2023). Explaining agent behavior with large language models. arXiv preprint arXiv:2309.10346, .
Zhang, D., Huang, X., Zhou, D., Li, Y., & Ouyang, W. (2024). Accessing gpt-4 level mathematical olympiad solutions via monte carlo tree self-refine with llama-3 8b. arXiv preprint arXiv:2406.07394, .
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., Liu, Z., Liu, P., Nie, J.-Y., & Wen, J.-R. (2025). A survey of large language models.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems NIPS ’23. Red Hook, NY, USA: Curran Associates Inc.
Ziaei, F., & Ranjbar, M. (2023). A reinforcement learning algorithm for scheduling parallel processors with identical speedup functions. Machine Learning with Applications, 13, 100485.
Zytek, A., Pido, S., Alnegheimish, S., Berti-Equille, L., & Veeramachaneni, K. (2024a). Explingo: Explaining ai predictions using large language models. In 2024 IEEE International Conference on Big Data (BigData) (pp. 1197–1208). IEEE.
Zytek, A., Pidò, S., & Veeramachaneni, K. (2024b). Llms for xai: Future directions for explaining explanations. arXiv preprint arXiv:2405.06064, .