Automated REST API Black-box Test Generation in Practice: An Experience Report from Industry

Automated black-box test generation for REST APIs has shown promise in research, yet industrial adoption remains limited. This paper presents an experience report on integrating two state-of-the-art REST API test generation tools, RestTestGen and DeepREST, into the quality assurance workflow of a space solutions company, to test a satellite connectivity system’s REST APIs comprising 204 API operations across five services implemented in Java, Python, and C++.
Empirical evaluation on production-like environments revealed high effectiveness, with median 74% operation coverage (151/204 operations) and detection of 21 unique previously unknown faults (e.g., missing input validation, database errors) in four out of five services, confirmed by developers. Both tools executed efficiently on commodity hardware, completing in under 10 minutes for most APIs (median CPU <100%, memory <1 GiB).
The company’s software engineers appreciated the tools’ ease of use, zero-setup Docker integration, and value for edge/negative scenarios, viewing them as complementary to manual tests rather than replacements. Key lessons include: automated tests complement manual ones by addressing negative scenarios, edge cases, and simple business logic; strict company policies hinder adoption of academic tools; and terminology mismatches between academic tools and practitioner expectations cause confusion.
These lessons pave the way to making academic tools more useful in industrial settings and may steer future research on REST API testing in industry.