A survey of flaky tests

O Parry, GM Kapfhammer, M Hilton… - ACM Transactions on …, 2021 - dl.acm.org
Tests that fail inconsistently, without changes to the code under test, are described as flaky.
Flaky tests do not give a clear indication of the presence of software bugs and thus limit the …

Context-aware code change embedding for better patch correctness assessment

B Lin, S Wang, M Wen, X Mao - ACM Transactions on Software …, 2022 - dl.acm.org
Despite the capability in successfully fixing more and more real-world bugs, existing
Automated Program Repair (APR) techniques are still challenged by the long-standing …

Flakify: A black-box, language model-based predictor for flaky tests

S Fatima, TA Ghaleb, L Briand - IEEE Transactions on Software …, 2022 - ieeexplore.ieee.org
Software testing assures that code changes do not adversely affect existing functionality.
However, a test case can be flaky, ie, passing and failing across executions, even for the …

A survey of challenges in spectrum-based software fault localization

QI Sarhan, Á Beszédes - IEEE Access, 2022 - ieeexplore.ieee.org
In software debugging, fault localization is the most difficult, expensive, tedious, and time-
consuming task, particularly for large-scale software systems. This is due to the fact that it …

Empirically evaluating flaky test detection techniques combining test case rerunning and machine learning models

O Parry, GM Kapfhammer, M Hilton… - Empirical Software …, 2023 - Springer
A flaky test is a test case whose outcome changes without modification to the code of the test
case or the program under test. These tests disrupt continuous integration, cause a loss of …

Evaluating classifiers in SE research: the ECSER pipeline and two replication studies

D Dell'Anna, FB Aydemir, F Dalpiaz - Empirical Software Engineering, 2023 - Springer
Context Automated classifiers, often based on machine learning (ML), are increasingly used
in software engineering (SE) for labelling previously unseen SE data. Researchers have …

A survey on how test flakiness affects developers and what support they need to address it

M Gruber, G Fraser - 2022 IEEE Conference on Software …, 2022 - ieeexplore.ieee.org
Non-deterministically passing and failing test cases, so-called flaky tests, have recently
become a focus area of software engineering research. While this research focus has been …

A qualitative study on the sources, impacts, and mitigation strategies of flaky tests

S Habchi, G Haben, M Papadakis… - … IEEE Conference on …, 2022 - ieeexplore.ieee.org
Test flakiness forms a major testing concern. Flaky tests manifest non-deterministic
outcomes that cripple continu-ous integration and lead developers to investigate false alerts …

Preempting flaky tests via non-idempotent-outcome tests

A Wei, P Yi, Z Li, T Xie, D Marinov, W Lam - Proceedings of the 44th …, 2022 - dl.acm.org
Regression testing can greatly help in software development, but it can be seriously
undermined by flaky tests, which can both pass and fail, seemingly nondeterministically, on …

When life gives you oranges: detecting and diagnosing intermittent job failures at mozilla

J Lampel, S Just, S Apel, A Zeller - Proceedings of the 29th ACM Joint …, 2021 - dl.acm.org
Continuous delivery of cloud systems requires constant running of jobs (build processes,
tests, etc.). One issue that plagues this continuous integration (CI) process are intermittent …