Towards Generating Executable Metamorphic Relations Using Large Language Models

Metamorphic testing (MT) has proven to be a successful solution to automating
testing and addressing the oracle problem. However, it entails manually
deriving metamorphic relations (MRs) and converting them into an executable
form; these steps are time-consuming and may prevent the adoption of MT. In
this paper, we propose an approach for automatically deriving executable MRs
(EMRs) from requirements using large language models (LLMs). Instead of merely
asking the LLM to produce EMRs, our approach relies on a few-shot prompting
strategy to instruct the LLM to perform activities in the MT process, by
providing requirements and API specifications, as one would do with software
engineers. To assess the feasibility of our approach, we conducted a
questionnaire-based survey in collaboration with Siemens Industry Software, a
worldwide leader in providing industry software and services, focusing on four
of their software applications. Additionally, we evaluated the accuracy of the
generated EMRs for a Web application. The outcomes of our study are highly
promising, as they demonstrate the capability of our approach to generate MRs
and EMRs that are both comprehensible and pertinent for testing purposes.