Natural Language to Code: How Far Are We?

A longstanding dream in software engineering research is to devise
e ective approaches for automating development tasks based on
developers’ informally-speci ed intentions. Such intentions are
generally in the form of natural language descriptions. In recent
literature, a number of approaches have been proposed to automate tasks such as code search and even code generation based on
natural language inputs. While these approaches vary in terms of
technical designs, their objective is the same: transforming a developer’s intention into source code. The literature, however, lacks a
comprehensive understanding towards the e ectiveness of existing techniques as well as their complementarity to each other. We
propose to ll this gap through a large-scale empirical study where
we systematically evaluate natural language to code techniques.
Speci cally, we consider six state-of-the-art techniques targeting
code search, and four targeting code generation. Through extensive
evaluations on a dataset of 22K+ natural language queries, our study
reveals the following major ndings: (1) code search techniques
based on model pre-training are so far the most e ective while
code generation techniques can also provide promising results;
(2) complementarity widely exists among the existing techniques;
and (3) combining the ten techniques together can enhance the performance for 35% compared with the most e ective standalone
technique. Finally, we propose a post-processing strategy to automatically integrate di erent techniques based on their generated
code. Experimental results show that our devised strategy is both
e ective and extensible.