What is Wrong with Topic Modeling? (and How to Fix it Using Search-based Software Engineering)
Context Topic modeling finds human-readable structures in unstructured textual data. A widely used topic modeling technique is Latent Dirichlet allocation. When running on different datasets, LDA suffers from “order effects”, i.e., different topics are generated if the order of training data is shuffled. Such order effects introduce a systematic error for any study. This error can relate to misleading results; specifically, inaccurate topic descriptions and a reduction in the efficacy of text mining classification results.
Objective To provide a method in which distributions generated by LDA are more stable and can be used for further analysis.
Method We use LDADE, a search-based software engineering tool which uses Differential Evolution (DE) to tune the LDA’s parameters. LDADE is evaluated on data from a programmer information exchange site (Stackoverflow), title and abstract text of thousands of Software Engineering (SE) papers, and software defect reports from NASA. Results were collected across different implementations of LDA (Python+Scikit-Learn, Scala+Spark) across Linux platform and for different kinds of LDAs (VEM, Gibbs sampling). Results were scored via topic stability and text mining classification accuracy.
Results In all treatments: (i) standard LDA exhibits very large topic instability; (ii) LDADE’s tunings dramatically reduce cluster instability; (iii) LDADE also leads to improved performances for supervised as well as unsupervised learning.
Conclusion Due to topic instability, using standard LDA with its “off-the-shelf” settings should now be depreciated. Also, in future, we should require SE papers that use LDA to test and (if needed) mitigate LDA topic instability. Finally, LDADE is a candidate technology for effectively and efficiently reducing that instability.
Thu 14 NovDisplayed time zone: Tijuana, Baja California change
16:00 - 17:40
Software DevelopmentResearch Papers / Journal First Presentations / Industry Showcase / Demonstrations at Hillcrest
Chair(s): Hitesh Sajnani Microsoft
|What is Wrong with Topic Modeling? (and How to Fix it Using Search-based Software Engineering)|
Journal First Presentations
Amritanshu Agrawal Wayfair, Wei Fu Department of Computer Science, North Carolina State University, Tim Menzies North Carolina State UniversityLink to publication
|Cautious Adaptation of Defiant Components|
Paulo Maia State University of Ceará, Lucas Vieira State University of Ceará, Matheus Chagas State University of Ceará, Yijun Yu The Open University, UK, Andrea Zisman The Open University, Bashar Nuseibeh The Open University (UK) & Lero (Ireland)
|Better Development of Safety Critical Systems:Chinese High Speed Railway System Development Experience Report|
Zhiwei Wu East China Normal University, Jing Liu East China Normal University, Xiang Chen CASCO Signal Ltd.
|Active Hotspot: An Issue-Oriented Model to Monitor Software Evolution and Degradation|
Qiong Feng Drexel University, Yuanfang Cai Drexel University, Rick Kazman University of Hawai‘i at Mānoa, Di Cui Xi'an Jiaotong University, Ting Liu Xi'an Jiaotong University, Hongzhou Fang Drexel University
|Automated Trainability Evaluation for Smart Software Functions|
Ilias Gerostathopoulos Technical University of Munich, Stefan Kugele Technical University of Munich, Christoph Segler BMW Group Research, New Technologies, Innovations, Tomas Bures Charles University, Czech Republic, Alois Knoll Technical University of MunichPre-print
|Lancer: Your Code Tell Me What You Need|
Shufan Zhou School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Beijun Shen School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Hao Zhong Shanghai Jiao Tong University