Making Sense of Failure Logs in an Industrial DevOps Environment

Abstract

Processing and reviewing nightly test execution failure logs for large industrial systems is a tedious activity. Furthermore, multiple failures might share one root/common cause during test execution sessions, and the review might therefore require redundant efforts. This paper presents the Log-Grouper approach for automated grouping of failure logs to aid root/common cause analysis and for enabling the processing of each log group as a batch. LogGrouper uses state-of-art natural language processing and clustering approaches to achieve meaningful log grouping. The approach is evaluated in an industrial setting in both a qualitative and quantitative manner. Results show that LogGrouper produces good quality groupings in terms of our two evaluation metrics (Silhouette Coefficient and Calinski-Harabasz Index) for clustering quality. The qualitative evaluation shows that experts perceive the groups as useful, and the groups are seen as an initial pointer for root cause analysis and failure assignment.

Experiences and challenges from developing cyber‐physical systems in industry‐academia collaboration

“…First, there has been a close collaboration with some partners, with extended data sharing and regular meetings, leading to AI-based tools that might fit Westermo's needs quite closely. One example is the LogGrouper tool 45 that uses large amounts of test execution logs and the test results database to cluster failing tests where one can expect one root cause to have triggered several failing tests. For this collaboration, Westermo has created personal non-disclosure agreements with individuals from the partner organization, in addition to the AIDOaRt-level contracts.…”