Write a Blog >>
ASE 2021
Sun 14 - Sat 20 November 2021 Australia
Thu 18 Nov 2021 21:20 - 21:40 at Kangaroo - Learning Applications Chair(s): Michael Pradel

Deep neural networks (DNNs) are being increasingly deployed as integral parts of software systems. However, due to the complex interconnections among hidden layers and massive hyperparameters, DNNs require being trained using a large number of labeled inputs, which calls for extensive human effort for collecting and labeling data. Spontaneously, to alleviate this growing demand, a surge of state-of-the-art studies comes up with different metrics to select a small yet informative dataset for the model training. These research works have demonstrated that DNN models can achieve competitive performance using a carefully selected small set of data. However, the literature lacks proper investigation of the limitations of data selection metrics, which is crucial to apply them in practice. In this paper, we fill this gap and conduct an extensive empirical study to explore the limits of selection metrics. Our study involves 15 selection metrics evaluated over 5 datasets (2 image classification tasks and 3 text classification tasks), 10 DNN architectures, and 20 labeling budgets (ratio of training data being labeled). Our findings reveal that, while selection metrics are usually effective in producing accurate models, they may induce a loss of model robustness (against adversarial examples) and resilience to compression. Overall, we demonstrate the existence of a trade-off between labeling effort and different model qualities. This paves the way for future research in devising selection metrics considering multiple quality criteria.

Thu 18 Nov

Displayed time zone: Hobart change

21:00 - 22:00
Learning ApplicationsResearch Papers / Tool Demonstrations / Journal-first Papers at Kangaroo
Chair(s): Michael Pradel University of Stuttgart
Deep GUI: Black-box GUI Input Generation with Deep Learning
Research Papers
Faraz YazdaniBanafsheDaragh University of California, Irvine, Sam Malek University of California at Irvine, USA
Towards Exploring the Limitations of Active Learning: An Empirical Study
Research Papers
Qiang Hu University of Luxembourg, Yuejun GUo University of Luxembourg, Maxime Cordy University of Luxembourg, Luxembourg, Xiaofei Xie Kyushu University, Wei Ma University of Luxembourg, Mike Papadakis University of Luxembourg, Luxembourg, Yves Le Traon University of Luxembourg, Luxembourg
Machine Learning based Success Prediction for Crowdsourcing Software Projects
Journal-first Papers
Inam Illahi Beijing Institute of Technology, Hui Liu Beijing Institute of Technology, Qasim Umer Beijing Institute of Technology, Nan Niu University of Cincinnati
SoManyConflicts: Resolve Many Merge Conflicts Interactively and Systematically
Tool Demonstrations
Bo Shen Peking University, Wei Zhang Peking University, Ailun Yu Peking University, Yifan Shi Peking University, Haiyan Zhao Peking University, Zhi Jin Peking University