Weblog and short text feature extraction and impact on categorisation

Abstract

The characterisation and categorisation of weblogs and other short texts has become an important research theme in the areas of topic/trend detection, and pattern recognition, amongst others. The value of analysing and characterising short text is to understand and identify the features that can identify and distinguish them, thereby improving input to the classification process. In this research work, we analyse a large number of text features and establish which combinations are useful to discriminate between the different genres of short text. Having identified the most promising features, we then confirm our findings by performing the categorisation task using three approaches: the Gaussian and SVM classifiers and the K-means clustering algorithm. Several hundred combinations of features were analysed in order to identify the best combinations and the results confirmed the observations made. The novel aspect of our work is the detection of the best combination of individual metrics which are identified as potential features to be used for the categorisation process.

"8 Amazing Secrets for Getting More Clicks": Detecting Clickbaits in News Streams Using Article Informality

“…To capture this difference in informality, we compute the following scores from the webpages and use them as features. These scores are indicative of the readability/informality level of a text and have been used previously to measure the informality level and reading difficulty of text (Miltsakaki and Troutt 2008;Mosquera and Moreda 2012;Pérez Téllez et al 2014;Lahiri, Mitra, and Lu 2011) CLScore, LIX and RIX are used to gauge the reading difficulty of the text. CLScore and (discretized) RIX index output the approximate US grade level required to comprehend the text whereas (discretized) LIX outputs scores corresponding to five levels of readability: very easy (0-24), easy (25-34), standard (35-44), difficult (45-54) and very difficult (more than 55).…”

Exposing Knowledge: Providing a Real-Time View of the Domain Under Study for Students

“…Fernando et al (2014) identified various features that helped them in characterization and categorization of Weblogs and other short texts. Most of the features rely on the words (tokens) within the text [35]. For effective transformation and for representation, word frequencies must be normalized in terms of their frequency within a document and within the entire collection [29].…”

Τεχνικές Μηχανικής Μάθησης Για Την Ανίχνευση Ομοιότητας/Ανομοιότητας

“…Η διαδικασία της αναγνώρισης του συνόλου των κόμβων που έχουν τη μεγαλύτερη επιρροή σε έναν κόμβο για ένα δεδομένο θέμα εισάγεται στο [61], η οποία, τελικά, έχει ως αποτέλεσμα την εύρεση των κόμβων που αντιπροσωπεύουν την πηγή για κάθε θέμα. Στο [72], ο χαρακτηρισμός και η κατηγοριοποίηση ιστολογίων και άλλων σύντομων κειμένων βελτιώνεται με την ανάλυση και τον εντοπισμό των χαρακτηριστικών που μπορούν να τα διακρίνουν. Στο [78], οι συγγραφείς προτείνουν ένα μοντέλο το οποίο χρησιμοποιεί δύο μοντέλα λανθάνουσας ανάθεσης Dirichlet (LDA) για την ομαδοποίηση παρόμοιων τηλεοπτικών χρηστών και παρόμοιων περιγραφών τηλεοπτικών προγραμμάτων ταυτόχρονα.…”