Write a Blog >>
MSR 2021
Mon 17 - Wed 19 May 2021
co-located with ICSE 2021

Continuous Integration (CI) is the process of automatically compiling, building, and testing code changes in the hope of catching bugs as they are introduced into the code base. With bug fixing being a core and increasingly costly task in software development, the community has adopted CI to mitigate this issue and improve the quality of their software products. Bug fixing is a core task in software development and becomes increasingly costly over time. However, little is known about how effective CI is at detecting simple, single-statement bugs.

In this paper, we analyze the effectiveness of CI in 14 popular open source Java-based projects to warn about 318 single-statement bugs (SStuBs). We analyze the build status at the commits that introduce SStuBs and before the SStuBs were fixed. We then investigate how often CI indicates the presence of these bugs, through test failure. Our results show that only 2% of the commits that introduced SStuBs have builds with failed tests and 7.5% of builds before the fix reported test failures. Upon close manual inspection, we found that none of the failed builds actually captured SStuBs, indicating that CI is not the right medium to capture the SStuBs we studied. Our results suggest that developers should not reply on CI to catch SStuBs or increase their CI pipeline coverage to detect single-statement bugs.

Mon 17 May

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

17:00 - 17:50
Mining Challenge SessionMining Challenge / Technical Papers at MSR Room 1
Chair(s): Miltiadis Allamanis Microsoft Research, UK, Rafael-Michael Karampatsis The University of Edinburgh, Charles Sutton Google Research
17:01
2m
Welcome by the Mining Challenge Co-chairs
Mining Challenge
Miltiadis Allamanis Microsoft Research, UK, Rafael-Michael Karampatsis The University of Edinburgh, Charles Sutton Google Research
17:03
3m
Talk
A large-scale study on human-cloned changes for automated program repair
Mining Challenge
Fernanda Madeiral KTH Royal Institute of Technology, Thomas Durieux KTH Royal Institute of Technology, Sweden
Link to publication Pre-print
17:06
3m
Talk
Applying CodeBERT for Automated Program Repair of Java Simple Bugs
Mining Challenge
Ehsan Mashhadi University of Calgary, Hadi Hemmati University of Calgary
Pre-print Media Attached
17:09
3m
Talk
PySStuBs: Characterizing Single-Statement Bugs in Popular Open-Source Python Projects
Mining Challenge
Arthur Veloso Kamienski University of Alberta, Luisa Palechor University of Alberta, Abram Hindle University of Alberta, Cor-Paul Bezemer University of Alberta
Pre-print
17:12
3m
Talk
How Effective is Continuous Integration in Indicating Single-Statement Bugs?
Mining Challenge
Jasmine Latendresse Concordia University, Rabe Abdalkareem Queens University, Kingston, Canada, Diego Costa Concordia University, Canada, Emad Shihab Concordia University
Pre-print
17:15
3m
Talk
Mea culpa: How developers fix their own simple bugs differently from other developers
Mining Challenge
Wenhan Zhu University of Waterloo, Michael W. Godfrey University of Waterloo, Canada
Pre-print
17:18
3m
Talk
On the Distribution of "Simple Stupid Bugs" in Unit Test Files: An Exploratory Study
Mining Challenge
Anthony Peruma Rochester Institute of Technology, Christian D. Newman Rochester Institute of Technology
Pre-print Media Attached
17:21
3m
Talk
On the Rise and Fall of Simple Stupid Bugs: a Life-Cycle Analysis of SStuBs
Mining Challenge
Balázs Mosolygó University of Szeged, Norbert Vándor University of Szeged, Gabor Antal University of Szeged, Peter Hegedus University of Szeged
Pre-print
17:24
3m
Talk
On the Effectiveness of Deep Vulnerability Detectors to Simple Stupid Bug Detection
Mining Challenge
Jiayi Hua Beijing University of Posts and Telecommunications, Haoyu Wang Beijing University of Posts and Telecommunications
Pre-print
17:27
23m
Live Q&A
Discussions and Q&A
Technical Papers


Information for Participants