Comparing Manual Text Patterns and Machine Learning for Classification of E-Mails for Automatic Answering by a Government Agency

Abstract

Abstract. E-mails to government institutions as well as to large companies may contain a large proportion of queries that can be answered in a uniform way. We analysed and manually annotated 4,404 e-mails from citizens to the Swedish Social Insurance Agency, and compared two methods for detecting answerable e-mails: manually-created text patterns (rule-based) and machine learning-based methods. We found that the text pattern-based method gave much higher precision at 89 percent than the machine learning-based method that gave only 63 percent precision. The recall was slightly higher (66 percent) for the machine learning-based methods than for the text patterns (47 percent). We also found that 23 percent of the total e-mail flow was processed by the automatic e-mail answering system.

Automated email answering by text‐pattern matching: Performance and error analysis

“…An alternative to SVM could be Naïve Bayes, which is also implemented in WEKA. In a previously conducted test A‐B for SVM and Naïve Bayes (Dalianis et al, ), both techniques had similar performance whereas SVM had a slightly better precision; we prioritize precision. In some older experiments with answer‐specific email categorization (Busemann, Schmeier, & Arens, ; Scheffer, ), SVM outperformed Naïve Bayes.…”

“…Finally, we manually labelled each message according to its text category. Preprocessing of the messages is covered in Dalianis et al ().…”

“…Bayes (Dalianis et al, 2011), both techniques had similar performance whereas SVM had a slightly better precision; we prioritize precision. In some…”

“…As the next step, we compared our text‐pattern matching technique with machine‐learning‐based text categorization and the bag‐of‐words text representation (Dalianis, Sjöbergh, & Sneiders, ). Such comparison was particularly interesting because email answering requires narrow answer‐oriented text categories whereas the traditional text categories are broad and topic oriented.…”

Learning Regular Expressions for Interpretable Medical Text Classification Using a Pool-based Simulated Annealing Approach

“…Neural network based approaches are strong alternatives but usually less interpretable because those black box models cannot be logically explained [15]. In addition, those black box models cannot be quickly modified except retraining models [16]. To address those difficult issues discussed above, some related work has been done by using regular expressions for classification tasks, and some autogenerated regular expressions can be effectively used to solve the classification problems as an interpretable way.…”