Automated REST API Black-box Test Generation in Practice: An Experience Report from Industry

CORRADINI, Davide; SHIN, Seung Yeob; BIANCULLI, Domenico

doi:10.1109/ICST69053.2026.00064

Download

Paper published in a book (Scientific congresses, symposiums and conference proceedings)

Automated REST API Black-box Test Generation in Practice: An Experience Report from Industry

CORRADINI, Davide; SHIN, Seung Yeob; BIANCULLI, Domenico

2026 • In Proceedings of ICST 2026

Peer reviewed

Permalink
https://hdl.handle.net/10993/68146

DOI
10.1109/ICST69053.2026.00064

Files (1)Send to Details Statistics Bibliography Similar publications

Files

Full Text

ICST-2026-REST-test-generation-in-practice.pdf

Author postprint (286.81 kB)

Download

All documents in ORBilu are protected by a user license.

Send to

RIS BibTex APA Chicago Permalink X Linkedin

Details

Keywords :

Software testing; Test generation; REST APIs; Industrial experience report; Software engineering

Abstract :

[en] Automated black-box test generation for REST APIs has shown promise in research, yet industrial adoption remains limited. This paper presents an experience report on integrating two state-of-the-art REST API test generation tools, RestTestGen and DeepREST, into the quality assurance workflow of a space solutions company, to test a satellite connectivity system’s REST APIs comprising 204 API operations across five services implemented in Java, Python, and C++. Empirical evaluation on production-like environments revealed high effectiveness, with median 74% operation coverage (151/204 operations) and detection of 21 unique previously unknown faults (e.g., missing input validation, database errors) in four out of five services, confirmed by developers. Both tools executed efficiently on commodity hardware, completing in under 10 minutes for most APIs (median CPU <100%, memory <1 GiB). The company’s software engineers appreciated the tools’ ease of use, zero-setup Docker integration, and value for edge/negative scenarios, viewing them as complementary to manual tests rather than replacements. Key lessons include: automated tests complement manual ones by addressing negative scenarios, edge cases, and simple business logic; strict company policies hinder adoption of academic tools; and terminology mismatches between academic tools and practitioner expectations cause confusion. These lessons pave the way to making academic tools more useful in industrial settings and may steer future research on REST API testing in industry.

Research center :

Interdisciplinary Centre for Security, Reliability and Trust (SnT) > SVV - Software Verification and Validation

Disciplines :

Computer science

Author, co-author :

CORRADINI, Davide ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV

SHIN, Seung Yeob ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV

BIANCULLI, Domenico ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV

External co-authors :

Language :

English

Title :

Automated REST API Black-box Test Generation in Practice: An Experience Report from Industry

Publication date :

July 2026

Event name :

19th IEEE International Conference on Software Testing, Verification and Validation (ICST)

Event place :

Daejeon, South Korea

Event date :

from 18 to 22 May 2026

Audience :

International

Main work title :

Proceedings of ICST 2026

Publisher :

Institute of Electrical and Electronics Engineers (IEEE)

Pages :

453-463

Peer review/Selection committee :

Peer reviewed

FnR Project :

FNR14016225 - INSTRUCT - Integrated Satellite-terrestrial Systems For Ubiquitous Beyond 5g Communications, 2020 (01/10/2020-30/09/2026) - Symeon Chatzinotas

Funding number :

IPBG19/14016225/INSTRUCT

Funding text :

This work has received funding from SES and the Luxembourg National Research Fund under the Industrial Partnership Block Grant (IPBG), ref. IPBG19/14016225/INSTRUCT.

Available on ORBilu :

since 03 April 2026

Statistics

Number of views

239 (13 by Unilu)

Number of downloads

160 (7 by Unilu)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

OpenAlex citations

Bibliography

C. Rodríguez, M. Baez, F. Daniel, F. Casati, J. C. Trabucco, L. Canali, and G. Percannella, "REST APIs: A large-scale analysis of compliance with principles and best practices, " in International conference on web engineering. Springer, 2016, pp. 21-39.
A. Neumann, N. Laranjeiro, and J. Bernardino, "An analysis of public REST web service APIs, " IEEE Transactions on Services Computing, vol. 14, no. 4, pp. 957-970, 2018.
A. Premchand and A. Choudhry, "Open banking & APIs for transformation in banking, " in 2018 international conference on communication, computing and internet of things (IC3IoT). IEEE, 2018, pp. 25-29.
H. Sartaj, S. Ali, T.Yue, and K. Moberg, "Testing real-world healthcare IoT application: Experiences and lessons learned, " in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023, pp. 2044-2049.
K. Deepika, R. R. Sankar, and A. Satyanarayana, "RESTful API integrations in telecom billing system management, " in Proceedings of the International Conference on Cognitive and Intelligent Computing: ICCIC 2021, Volume 1. Springer, 2022, pp. 323-329.
V. Atlidakis, P. Godefroid, and M. Polishchuk, "RESTler: Stateful REST API fuzzing, " in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 2019, pp. 748-758.
A. Arcuri, "RESTful API automated test case generation with EvoMaster, " ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 28, no. 1, pp. 1-37, 2019.
"Automated black-and white-box testing of RESTful APIs with EvoMaster, " IEEE Software, vol. 38, no. 3, pp. 72-78, 2020.
V. Atlidakis, R. Geambasu, P. Godefroid, M. Polishchuk, and B. Ray, "Pythia: Grammar-based fuzzing of REST APIs with coverage-guided feedback and learning-based mutations, " arXiv preprint arXiv:2005.11498, 2020.
S. Karlsson, A. Čaušević, and D. Sundmark, "QuickREST: Propertybased test generation of OpenAPI-described RESTful APIs, " in 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST). IEEE, 2020, pp. 131-141.
A. Martin-Lopez, S. Segura, and A. Ruiz-Cortés, "RESTest: Blackbox constraint-based testing of RESTful web APIs, " in International Conference on Service-Oriented Computing. Springer, 2020, pp. 459-475.
"RESTest: automated black-box testing of RESTful web APIs, " in Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2021, pp. 682-685.
N. Laranjeiro, J. Agnelo, and J. Bernardino, "A black box tool for robustness testing of REST services, " IEEE Access, vol. 9, pp. 24 738-24 754, 2021.
J. C. Alonso, A. Martin-Lopez, S. Segura, J. M. Garcia, and A. Ruiz-Cortes, "ARTE: Automated generation of realistic test inputs for web APIs, " IEEE Transactions on Software Engineering, vol. 49, no. 1, pp. 348-363, 2022.
D. Corradini, A. Zampieri, M. Pasqua, E. Viglianisi, M. Dallago, and M. Ceccato, "Automated black-box testing of nominal and error scenarios in RESTful APIs, " Software Testing, Verification and Reliability, vol. 32, no. 5, p. e1808, 2022.
H. Wu, L. Xu, X. Niu, and C. Nie, "Combinatorial testing of RESTful APIs, " in Proceedings of the 44th International Conference on Software Engineering, 2022, pp. 426-437.
Z. Hatfield-Dodds and D. Dygalo, "Deriving semantics-aware fuzzers from web API schemas, " in Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, 2022, pp. 345-346.
Y. Liu, Y. Li, G. Deng, Y. Liu, R. Wan, R. Wu, D. Ji, S. Xu, and M. Bao, "Morest: Model-based RESTful API testing with execution feedback, " in Proceedings of the 44th International Conference on Software Engineering, 2022, pp. 1406-1417.
M. Kim, S. Sinha, and A. Orso, "Adaptive REST API testing with reinforcement learning, " in 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2023, pp. 446-458.
D. Corradini, Z. Montolli, M. Pasqua, and M. Ceccato, "DeepREST: Automated test case generation for REST APIs exploiting deep reinforcement learning, " in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, 2024, pp. 1383-1394.
T.-Q. Nguyen, N.-H. Cong, N.-M. Quach, H. D. Vo, and S. Nguyen, "Reinforcement learning-based REST API testing with multi-coverage, " arXiv preprint arXiv:2410.15399, 2024.
M. Kim, S. Sinha, and A. Orso, "LlamaRestTest: Effective REST API testing with small language models, " Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, pp. 465-488, 2025.
T. Stennett, M. Kim, S. Sinha, and A. Orso, "AutoRestTest: A tool for automated REST API testing using LLMs and MARL, " in 2025 IEEE/ACM 47th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 2025, pp. 21-24.
K. Zhang, C. Zhang, C. Wang, C. Zhang, Y. Wu, Z. Xing, Y. Liu, Q. Li, and X. Peng, "LogiAgent: Automated logical testing for REST systems with LLM-based multi-agents, " arXiv preprint arXiv:2503.15079, 2025.
Y. Liu, Y. Li, Y. Liu, R.Wan, R.Wu, and Q. Liu, "Morest: industry practice of automatic RESTful API testing, " in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022, pp. 1-5.
M. Zhang, A. Arcuri, Y. Li, Y. Liu, and K. Xue, "White-box fuzzing RPCbased APIs with EvoMaster: An industrial case study, " ACM Transactions on Software Engineering and Methodology, vol. 32, no. 5, pp. 1-38, 2023.
M. Zhang, A. Arcuri, P. Teng, K. Xue, and W. Wang, "Seeding and mocking in white-box fuzzing enterprise RPC APIs: An industrial case study, " in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, 2024, pp. 2024-2034.
M. Zhang, A. Arcuri, Y. Li, Y. Liu, K.Xue, Z.Wang, J. Huo, and W. Huang, "Fuzzing microservices: A series of user studies in industry on industrial systems with EvoMaster, " Science of Computer Programming, p. 103322, 2025.
A. Poth, O. Rrjolli, and A. Arcuri, "Technology adoption performance evaluation applied to testing industrial REST APIs, " Automated Software Engineering, vol. 32, no. 1, p. 5, 2025.
A. Arcuri, A. Poth, and O. Rrjolli, "Introducing black-box fuzz testing for REST APIs in industry: Challenges and solutions, " in 2025 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 2025, pp. 382-393.
X. Yu, L. Liu, X. Hu, J. Keung, X. Xia, and D. Lo, "Practitioners' expectations on automated test generation, " in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 1618-1630.
R. T. Fielding, Architectural styles and the design of network-based software architectures. University of California, Irvine, 2000.
A. Lawi, B. L. Panggabean, and T. Yoshida, "Evaluating GraphQL and REST API services performance in a massive and intensive accessible information system, " Computers, vol. 10, no. 11, p. 138, 2021.
D. Corradini, A. Zampieri, M. Pasqua, and M. Ceccato, "RestTestGen: An extensible framework for automated black-box testing of RESTful APIs, " in 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2022, pp. 504-508.
O. Al-Debagy and P. Martinek, "A metrics framework for evaluating microservices architecture designs, " Journal of Web Engineering, vol. 19, no. 3-4, pp. 341-370, 2020.
A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv et al., "Qwen3 technical report, " arXiv preprint arXiv:2505.09388, 2025.
A. Golmohammadi, M. Zhang, and A. Arcuri, "Testing RESTful APIs: A survey, " ACM Transactions on Software Engineering and Methodology, vol. 33, no. 1, pp. 1-41, 2023.
A. Martin-Lopez, S. Segura, and A. Ruiz-Cortés, "Online testing of RESTful APIs: Promises and challenges, " in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022, pp. 408-420.