A Log-Based Testing Approach for Detecting Faults Caused by Incorrect Assumptions About the Environment

Jeong, Sooyong; Jha, Ajay Kumar; Shin, Youngsul; Lee, Woojin

doi:10.1587/transinf.2019edl8149

Cited by 3 publications

(1 citation statement)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The approach employs search-based testing, machine learning, and model checking techniques. Jeong et al [15] also focused on environment assumptions, and they proposed a log-based testing approach to identify faults caused by invalid environment assumptions in model-based development. Li et al [19] conducted an experiment to evaluate the performance of seven machine learning methods (e.g., Support Vector Machine, Logistic Regression, and Perceptron) in automatically classifying software development assumptions.…”

Section: Assumptions In Software Developmentmentioning

confidence: 99%

Self-Claimed Assumptions in Deep Learning Frameworks: An Exploratory Study

Yang

Liang

et al. 2021

Evaluation and Assessment in Software Engineering

View full text Add to dashboard Cite

Deep learning (DL) frameworks have been extensively designed, implemented, and used in software projects across many domains. However, due to the lack of knowledge or information, time pressure, complex context, etc., various uncertainties emerge during the development, leading to assumptions made in DL frameworks. Though not all the assumptions are negative to the frameworks, being unaware of certain assumptions can result in critical problems (e.g., system vulnerability and failures, inconsistencies, and increased cost). As the first step of addressing the critical problems, there is a need to explore and understand the assumptions made in DL frameworks. To this end, we conducted an exploratory study to understand self-claimed assumptions (SCAs) about their distribution, classification, and impacts using code comments from nine popular DL framework projects on GitHub. The results are that: (1) 3,084 SCAs are scattered across 1,775 files in the nine DL frameworks, ranging from 1,460 (TensorFlow) to 8 (Keras) SCAs. (2) There are four types of validity of SCAs: Valid SCA, Invalid SCA, Conditional SCA, and Unknown SCA, and four types of SCAs based on their content: Configuration and Context SCA, Design SCA, Tensor and Variable SCA, and Miscellaneous SCA. (3) Both valid and invalid SCAs may have an impact within a specific scope (e.g., in a function) on the DL frameworks. Certain technical debt is induced when making SCAs. There are source code written and decisions made based on SCAs. This is the first study on investigating SCAs in DL frameworks, which helps researchers and practitioners to get a comprehensive understanding on the assumptions made. We also provide the first dataset of SCAs for further research and practice in this area.

show abstract

Section: Assumptions In Software Developmentmentioning

confidence: 99%

Self-Claimed Assumptions in Deep Learning Frameworks: An Exploratory Study

Yang

Liang

et al. 2021

Evaluation and Assessment in Software Engineering

View full text Add to dashboard Cite

show abstract

Impact of log parsing on deep learning-based anomaly detection

Khan,

Shin,

Bianculli

et al. 2024

Empir Software Eng

View full text Add to dashboard Cite

Software systems log massive amounts of data, recording important runtime information. Such logs are used, for example, for log-based anomaly detection, which aims to automatically detect abnormal behaviors of the system under analysis by processing the information recorded in its logs. Many log-based anomaly detection techniques based on deep learning models include a pre-processing step called log parsing. However, understanding the impact of log parsing on the accuracy of anomaly detection techniques has received surprisingly little attention so far. Investigating what are the key properties log parsing techniques should ideally have to help anomaly detection is therefore warranted. In this paper, we report on a comprehensive empirical study on the impact of log parsing on anomaly detection accuracy, using 13 log parsing techniques, seven anomly detection techniques (five based on deep learning and two based on traditional machine learning) on three publicly available log datasets. Our empirical results show that, despite what is widely assumed, there is no strong correlation between log parsing accuracy and anomaly detection accuracy, regardless of the metric used for measuring log parsing accuracy. Moreover, we experimentally confirm existing theoretical results showing that it is a property that we refer to as distinguishability in log parsing results—as opposed to their accuracy—that plays an essential role in achieving accurate anomaly detection.

show abstract