Write a Blog >>
ICSE 2021
Mon 17 May - Sat 5 June 2021
Thu 27 May 2021 15:45 - 16:05 at Blended Sessions Room 4 - 3.3.4. Testing: Flaky Tests Chair(s): José Miguel Rojas
Fri 28 May 2021 03:45 - 04:05 at Blended Sessions Room 4 - 3.3.4. Testing: Flaky Tests

Flaky tests have gained attention from the research community in recent years and with good reason. These tests lead to wasted time and resources and reduce the reliability of the test suites and build systems they affect. However, most of the existing works on flaky tests focus exclusively on traditional unit tests. This ignores UI tests that have larger input spaces and more diverse running conditions than traditional unit tests. In addition, UI tests tend to be more complex and resource-heavy, making them unsuited for detection techniques involving rerunning test suites multiple times. In this paper, we perform a study on UI flaky tests. We analyze 235 flaky UI test samples found in 62 projects from both web and Android environments. We identify the common underlying root causes of flakiness in the UI tests, the strategies used to manifest the flaky behavior, and the fixing strategies used to remedy flaky UI tests. The findings made in this work can provide a foundation for the development of detection and prevention techniques for flakiness arising in UI tests.

Thu 27 May

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

15:05 - 16:05
3.3.4. Testing: Flaky TestsTechnical Track / Journal-First Papers at Blended Sessions Room 4 +12h
Chair(s): José Miguel Rojas University of Leicester, UK
15:05
20m
Paper
Quantifying, Characterizing, and Mitigating Flakily Covered Program ElementsJournal-First
Journal-First Papers
Pre-print Media Attached
15:25
20m
Paper
FlakeFlagger: Predicting Flakiness Without Rerunning TestsArtifact ReusableTechnical TrackArtifact Available
Technical Track
Abdulrahman Alshammari George Mason University, Christopher Morris Carnegie Mellon University, Michael Hilton Carnegie Mellon University, USA, Jonathan Bell Northeastern University
Pre-print Media Attached
15:45
20m
Paper
An Empirical Analysis of UI-based Flaky TestsArtifact ReusableTechnical TrackArtifact Available
Technical Track
Alan Romano University at Buffalo, Zihe Song University of Texas at Dallas, Sampath Grandhi University of Texas at Dallas, Wei Yang University of Texas at Dallas, Weihang Wang University at Buffalo, SUNY
Pre-print Media Attached

Fri 28 May

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

03:05 - 04:05
03:05
20m
Paper
Quantifying, Characterizing, and Mitigating Flakily Covered Program ElementsJournal-First
Journal-First Papers
Pre-print Media Attached
03:25
20m
Paper
FlakeFlagger: Predicting Flakiness Without Rerunning TestsArtifact ReusableTechnical TrackArtifact Available
Technical Track
Abdulrahman Alshammari George Mason University, Christopher Morris Carnegie Mellon University, Michael Hilton Carnegie Mellon University, USA, Jonathan Bell Northeastern University
Pre-print Media Attached
03:45
20m
Paper
An Empirical Analysis of UI-based Flaky TestsArtifact ReusableTechnical TrackArtifact Available
Technical Track
Alan Romano University at Buffalo, Zihe Song University of Texas at Dallas, Sampath Grandhi University of Texas at Dallas, Wei Yang University of Texas at Dallas, Weihang Wang University at Buffalo, SUNY
Pre-print Media Attached