… Moreover, experiments on discriminative tasks, eg, program repair, clone detection, and …
in program understanding. Furthermore, analysis reveals that PLBART learns program syntax, …

… By using test cases, we provide a gold-standard metric for code generation quality. … measuring
performance in an accurate and rigorous manner. Using APPS, we find that programming …

… benchmark dataset for program understanding and genera… of 10 diversified programming
language understanding and … metric for the code completion scenario as it measures how …

… For each of these challenges, we explicitly did not evaluate metrics such as the participant’s
productivity, the quality of their solutions, or the time taken to produce them, as the focus of …

… This approach of calculating the F1-measure has been used in similar … metrics, we
only showcase the weighted F1-measure in the paper, whereas the precision, recall, and F1-measure …

… , such as software metrics and code metrics. For example, Kumar et al. [EL18] used a
metric-… Code comments are crucial to program comprehension. If code generation was to …

… Chapter 8, following the chapter on software design, steps back and discusses design
characteristics and metrics used in evaluating high-level and detailed designs. Chapter 11 …

… Software Engineering This book offers a practical approach to understanding, designing,
and building sound software … for quality software, and learn the basics of the most common …

… This section reviews quantitative metrics for measuring the quality of three types of
explanation methods: model-based explanations, attribution-based explanations, and example-…

… the pass@k metric, and explain its advantages over standard match-based metrics. Next, …
We argue that this metric should be applied to docstringconditional code generation as well. …