Extreme mutation testing in practice: An industrial case study
Mutation testing is used to evaluate the effectiveness of test suites. In recent years, a promising variation called extreme mutation testing emerged that is computationally less expensive. It identifies methods where their functionality can be entirely removed, and the test suite would not notice it, despite having coverage. These methods are called pseudo-tested. In this paper, we compare the execution and analysis times for traditional and extreme mutation testing and discuss what they mean in practice. We look at how extreme mutation testing impacts current software development practices and discuss open challenges that need to be addressed to foster industry adoption. For that, we conducted an industrial case study consisting of running traditional and extreme mutation testing in a large software project from the semiconductor industry that is covered by a test suite of more than 11,000 unit tests. In addition to that, we did a qualitative analysis of 25 pseudo-tested methods and interviewed two experienced developers to see how they write unit tests and gathered opinions on how useful the findings of extreme mutation testing are. Our results include execution times, scores, numbers of executed tests and mutators, reasons why methods are pseudo-tested, and an interview summary. We conclude that the shorter execution and analysis times are well noticeable in practice and show that extreme mutation testing supplements writing unit tests in conjunction with code coverage tools. We propose that pseudo-tested code should be highlighted in code coverage reports and that extreme mutation testing should be performed when writing unit tests rather than having a separate session to analyze the findings. For that, future research should investigate how to perform extreme mutation testing while writing unit tests such that the results are available fast enough but still meaningful.
Thu 20 MayDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
15:00 - 16:15 | Mutation Testing and DebuggingAST 2021 at AST Room Chair(s): Francisco Gortázar Universidad Rey Juan Carlos | ||
15:00 30mLong-paper | SuMo: A Mutation Strategy for Solidity Smart Contracts AST 2021 Pre-print Media Attached | ||
15:30 15mShort-paper | Extreme mutation testing in practice: An industrial case study AST 2021 Pre-print Media Attached | ||
15:45 30mLong-paper | Extending Hierarchical Delta Debugging with Hoisting AST 2021 Daniel Vince , Renáta Hodován University of Szeged, Hungary, Daniella Bársony University of Szeged, Department of Software Engineering, Ákos Kiss University of Szeged, Hungary Pre-print Media Attached |
Go directly to this room on Clowdr