Understanding and handling alert storm for online service systems

Zhao, Ning; Chen, Junjie; Peng, Xiao; Wang, Honglin; Wu, Xinya; Zhang, Yuanzong; Chen, Zikai; Zheng, Xiangzhong; Nie, Xin; Wang, Gang; Wu, Yingxin; Zhou, Fang; Zhang, Wenchi; Sui, Kaixin; Pei, Dan

doi:10.1145/3377813.3381363

Cited by 24 publications

(23 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hence, the feedback information obtained as a final result has tightened the connection between operations and development. There have been several attempts at addressing similar challenges using state of the art solutions based on deep learning [6,20,26], while our solution proposal reach promising results while keeping simplicity of the ML approach. The smart filter in the feedback loop improves the connection between operations and development but at the same time raises more challenges that need to be addressed in the future.…”

Section: Discussionmentioning

confidence: 91%

“…Alerts is another term used for denoting the same or similar events as alarms and according to Zhao et al [6], they represent a key source of anomalous events in operations. Zhao et al [6] reported an approach for handling alert storms consisting of alert storm detection using Extreme Value Theory (EVT), alert filtering using ML Isolation Forest method, alert clustering using Similarity Matrix Construction, and representative alert selection. Furthermore, Zhao et al [17] published another study on enhancing the quality of services by utilizing the monitoring data.…”

Section: Background and Related Workmentioning

confidence: 99%

“…In summary, operations data has been studied and analyzed for different purposes but still, there is more to be explored in DevOps contexts, to improve the feedback from operations to development. State of the art solutions [6,19,26] address relevant challenges in managing operations data. However, situations of alert flooding in DevOps environments are not extensively explored.…”

Section: Background and Related Workmentioning

confidence: 99%

“…Hence, we improve the feedback loop from operations to development by introducing a new element, a smart filter, for optimization of alert to noise ratio. In the design process, we considered the insights gained through interviews, results of the intensive discussions with the development team, and state of the art solutions for alert management [6,17].…”

Section: Research Approachmentioning

confidence: 99%

“…In the literature, there are examples of various methods for the analysis of operations data but only a few are addressing real industrial needs and challenges companies are facing in relation to the feedback from operations to development [6]. Consequently, there is a limited choice of potential solutions available in the literature for designing more context-specific solution designs based on the identified industrial needs.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Closing the Feedback Loop in DevOps Through Autonomous Monitors in Operations

2021

View full text Add to dashboard Cite

DevOps represent the tight connection between development and operations. To address challenges that arise on the borderline between development and operations, we conducted a study in collaboration with a Swedish company responsible for ticket management and sales in public transportation. The aim of our study was to explore and describe the existing DevOps environment, as well as to identify how the feedback from operations can be improved, specifically with respect to the alerts sent from system operations. Our study complies with the basic principles of the design science paradigm, such as understanding and improving design solutions in the specific areas of practice. Our diagnosis, based on qualitative data collected through interviews and observations, shows that alert flooding is a challenge in the feedback loop, i.e. too much signals from operations create noise in the feedback loop. Therefore, we design a solution to improve the alert management by optimizing when to raise alerts and accordingly introducing a new element in the feedback loop, a smart filter. Moreover, we implemented a prototype of the proposed solution design and showed that a tighter relation between operations and development can be achieved, using a hybrid method which combines rule-based and unsupervised machine learning for operations data analysis.

show abstract

Section: Discussionmentioning

confidence: 91%