IT Ticket Classification: The Simpler, the Better

Abstract

Recently, automatic classification of IT tickets has gained notable attention due to the increasing complexity of IT services deployed in enterprises. There are multiple discussions and no general opinion in the research and practitioners' community on the design of IT ticket classification tasks, specifically the choice of ticket text representation techniques and classification algorithms. Our study aims to investigate the core design elements of a typical IT ticket text classification pipeline. In particular, we compare the performance of TF-IDF and linguistic features-based text representations designed for ticket complexity prediction. We apply various classifiers, including kNN, its enhanced versions, decision trees, naïve Bayes, logistic regression, support vector machines, as well as semi-supervised techniques to predict the ticket class label of low, medium, or high complexity. Finally, we discuss the evaluation results and their practical implications. As our study shows, linguistic representation not only proves to be highly explainable but also demonstrates a substantial prediction quality increase over TF-IDF. Furthermore, our experiments evidence the importance of feature selection. We indicate that even simple algorithms can deliver high-quality prediction when using appropriate linguistic features.

Improving Support Ticket Systems Using Machine Learning: A Literature Review

“…When analyzing the 4 papers presenting DNNbased approaches [19,21,23,29], we found that two of those papers used specific LSTM-networks, which were both the best performing approaches in that papers [19,29]. Since two papers are not a sufficiently large sample, no general statement can be derived here.…”

“…Han and Sun [21] x Mour, Dey [22] x x Revina, Buza [23] x Xu, Mu [24] x x Al-Hawari and Barham [8] x…”

“…In addition, a trend can be identified over the years. According to Revina, Buza [23], the earliest approaches of support ticket classification were ruledbased approaches that were outperformed by RF and SVM. This probably is the reason that mostly in the years 2018 and 2019, RF and SVM were the approaches most implemented and evaluated and also best performing [2,6,8,9,30,36].…”

“…This probably is the reason that mostly in the years 2018 and 2019, RF and SVM were the approaches most implemented and evaluated and also best performing [2,6,8,9,30,36]. Since 2019, some authors also propose artificial-neuronal-networkbased solutions, because of the good performance of neuronal network approaches in other fields of ML [19,21,23,29]. While there is some promising evidence, that Deep Neuronal Network (DNN) based solutions might be able to outperform present ML approaches like RF or SVM [29] (reported accuracies: 77% [29], 83% [19], 99% [21]), this point seems not to be fully reached yet [23].…”

“…Since 2019, some authors also propose artificial-neuronal-networkbased solutions, because of the good performance of neuronal network approaches in other fields of ML [19,21,23,29]. While there is some promising evidence, that Deep Neuronal Network (DNN) based solutions might be able to outperform present ML approaches like RF or SVM [29] (reported accuracies: 77% [29], 83% [19], 99% [21]), this point seems not to be fully reached yet [23]. Revina, Buza [23] find in their conclusion (page 11) "that simple algorithms work well if using appropriate linguistic features" and can be equally performative.…”