Logram: Efficient Log Parsing Using n-Gram DictionariesJournal-First
Wed 26 May 2021 08:15 - 08:35 at Blended Sessions Room 1 - 1.5.1. Deep Neural Networks: General Issues
Software systems usually record important runtime information in their logs. Logs help practitioners understand system runtime behaviors and diagnose field failures. As logs are usually very large in size, automated log analysis is needed to assist practitioners in their software operation and maintenance efforts. Typically, the first step of automated log analysis is log parsing, i.e., converting unstructured raw logs into structured data. However, log parsing is challenging, because logs are produced by static templates in the source code (i.e., logging statements) yet the templates are usually inaccessible when parsing logs. Prior work proposed automated log parsing approaches that have achieved high accuracy. However, as the volume of logs grows rapidly in the era of cloud computing, efficiency becomes a major concern in log parsing. In this work, we propose an automated log parsing approach, Logram, which leverages n-gram dictionaries to achieve efficient log parsing. We evaluated Logram on 16 public log datasets and compared Logram with five state-of-the-art log parsing approaches. We found that Logram achieves a higher parsing accuracy than the best existing approaches (i.e., at least 10% higher, on average) and also outperforms these approaches in efficiency (i.e., 1.8 to 5.1 times faster than the second-fastest approaches in terms of end-to-end parsing time). Furthermore, we deployed Logram on Spark and we found that Logram scales out efficiently with the number of Spark nodes (e.g., with near-linear scalability for some logs) without sacrificing parsing accuracy. In addition, we demonstrated that Logram can support effective online parsing of logs, achieving similar parsing results and efficiency to the offline mode.
Tue 25 MayDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
19:35 - 20:55 | 1.5.1. Deep Neural Networks: General IssuesTechnical Track / Journal-First Papers / SEIP - Software Engineering in Practice at Blended Sessions Room 1 +12h Chair(s): Ignacio Panach Universidad de Valencia | ||
19:35 20mPaper | Asset Management in Machine Learning: A SurveySEIP SEIP - Software Engineering in Practice Samuel Idowu Chalmers | University of Gothenburg, Daniel Strüber Radboud University Nijmegen, Thorsten Berger Chalmers | University of Gothenburg Pre-print Media Attached | ||
19:55 20mPaper | An Empirical Study of Refactorings and Technical Debt in Machine Learning SystemsTechnical Track Technical Track Yiming Tang City University of New York (CUNY) Graduate Center, Raffi Khatchadourian CUNY Hunter College, Mehdi Bagherzadeh Oakland University, Rhia Singh City University of New York (CUNY) Macaulay Honors College, Ajani Stewart City University of New York (CUNY) Hunter College, Anita Raja City University of New York (CUNY) Hunter College Pre-print Media Attached | ||
20:15 20mPaper | Logram: Efficient Log Parsing Using n-Gram DictionariesJournal-First Journal-First Papers Hetong Dai Concordia University, Heng Li Polytechnique Montréal, Che-Shao Chen Concordia University, Weiyi Shang Concordia University, Tse-Hsun (Peter) Chen Concordia University DOI Pre-print Media Attached | ||
20:35 20mPaper | DeepLocalize: Fault Localization for Deep Neural NetworksTechnical Track Technical Track Mohammad Wardat Dept. of Computer Science, Iowa State University, Wei Le Dept. of Computer Science, Iowa State University, Hridesh Rajan Dept. of Computer Science, Iowa State University Pre-print Media Attached |
Wed 26 MayDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
07:35 - 08:55 | 1.5.1. Deep Neural Networks: General IssuesTechnical Track / SEIP - Software Engineering in Practice / Journal-First Papers at Blended Sessions Room 1 | ||
07:35 20mPaper | Asset Management in Machine Learning: A SurveySEIP SEIP - Software Engineering in Practice Samuel Idowu Chalmers | University of Gothenburg, Daniel Strüber Radboud University Nijmegen, Thorsten Berger Chalmers | University of Gothenburg Pre-print Media Attached | ||
07:55 20mPaper | An Empirical Study of Refactorings and Technical Debt in Machine Learning SystemsTechnical Track Technical Track Yiming Tang City University of New York (CUNY) Graduate Center, Raffi Khatchadourian CUNY Hunter College, Mehdi Bagherzadeh Oakland University, Rhia Singh City University of New York (CUNY) Macaulay Honors College, Ajani Stewart City University of New York (CUNY) Hunter College, Anita Raja City University of New York (CUNY) Hunter College Pre-print Media Attached | ||
08:15 20mPaper | Logram: Efficient Log Parsing Using n-Gram DictionariesJournal-First Journal-First Papers Hetong Dai Concordia University, Heng Li Polytechnique Montréal, Che-Shao Chen Concordia University, Weiyi Shang Concordia University, Tse-Hsun (Peter) Chen Concordia University DOI Pre-print Media Attached | ||
08:35 20mPaper | DeepLocalize: Fault Localization for Deep Neural NetworksTechnical Track Technical Track Mohammad Wardat Dept. of Computer Science, Iowa State University, Wei Le Dept. of Computer Science, Iowa State University, Hridesh Rajan Dept. of Computer Science, Iowa State University Pre-print Media Attached |