A survey of flaky tests
O Parry, GM Kapfhammer, M Hilton… - ACM Transactions on …, 2021 - dl.acm.org
Tests that fail inconsistently, without changes to the code under test, are described as flaky.
Flaky tests do not give a clear indication of the presence of software bugs and thus limit the …
Flaky tests do not give a clear indication of the presence of software bugs and thus limit the …
Context-aware code change embedding for better patch correctness assessment
Despite the capability in successfully fixing more and more real-world bugs, existing
Automated Program Repair (APR) techniques are still challenged by the long-standing …
Automated Program Repair (APR) techniques are still challenged by the long-standing …
Flakify: A black-box, language model-based predictor for flaky tests
Software testing assures that code changes do not adversely affect existing functionality.
However, a test case can be flaky, ie, passing and failing across executions, even for the …
However, a test case can be flaky, ie, passing and failing across executions, even for the …
A survey of challenges in spectrum-based software fault localization
QI Sarhan, Á Beszédes - IEEE Access, 2022 - ieeexplore.ieee.org
In software debugging, fault localization is the most difficult, expensive, tedious, and time-
consuming task, particularly for large-scale software systems. This is due to the fact that it …
consuming task, particularly for large-scale software systems. This is due to the fact that it …
Empirically evaluating flaky test detection techniques combining test case rerunning and machine learning models
O Parry, GM Kapfhammer, M Hilton… - Empirical Software …, 2023 - Springer
A flaky test is a test case whose outcome changes without modification to the code of the test
case or the program under test. These tests disrupt continuous integration, cause a loss of …
case or the program under test. These tests disrupt continuous integration, cause a loss of …
Evaluating classifiers in SE research: the ECSER pipeline and two replication studies
Context Automated classifiers, often based on machine learning (ML), are increasingly used
in software engineering (SE) for labelling previously unseen SE data. Researchers have …
in software engineering (SE) for labelling previously unseen SE data. Researchers have …
A survey on how test flakiness affects developers and what support they need to address it
Non-deterministically passing and failing test cases, so-called flaky tests, have recently
become a focus area of software engineering research. While this research focus has been …
become a focus area of software engineering research. While this research focus has been …
A qualitative study on the sources, impacts, and mitigation strategies of flaky tests
Test flakiness forms a major testing concern. Flaky tests manifest non-deterministic
outcomes that cripple continu-ous integration and lead developers to investigate false alerts …
outcomes that cripple continu-ous integration and lead developers to investigate false alerts …
Preempting flaky tests via non-idempotent-outcome tests
Regression testing can greatly help in software development, but it can be seriously
undermined by flaky tests, which can both pass and fail, seemingly nondeterministically, on …
undermined by flaky tests, which can both pass and fail, seemingly nondeterministically, on …
When life gives you oranges: detecting and diagnosing intermittent job failures at mozilla
Continuous delivery of cloud systems requires constant running of jobs (build processes,
tests, etc.). One issue that plagues this continuous integration (CI) process are intermittent …
tests, etc.). One issue that plagues this continuous integration (CI) process are intermittent …