A Varied Density-based Clustering Approach for Event Detection from Heterogeneous Twitter Data

Ghaemi, Zeinab; Farnaghi, Mahdi

doi:10.3390/ijgi8020082

Cited by 27 publications

(12 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although K-Means is very fast and mostly used for clustering, it requires one to define the number of clusters as a parameter to the algorithm. Among the existing clustering approaches, a density-based algorithm particularly DBSCAN (density-based spatial clustering with noise) and its variations, is more efficient for detecting clusters with arbitrary shapes from the noisy dataset where there is no prior knowledge about the number of clusters (Ghaemi and Farnaghi, 2019;Liu et al, 2007). Many improved versions of this algorithm are also available (such as NG-DBSCAN (Lulli et al, 2016)) to overcome the scalability issues of densitybased clustering, but they fail to address the ineffectiveness of density-based approaches in sparse data setting.…”

Section: Related Workmentioning

confidence: 99%

Intent Mining from past conversations for Conversational Agent

Chatterjee¹,

Sengupta²

2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

Conversational systems are of primary interest in the AI community. Organizations are increasingly using chatbot to provide round-the-clock support and to increase customer engagement. Many commercial bot building frameworks follow a standard approach that requires one to build and train an intent model to recognize user input. These frameworks require a collection of user utterances and corresponding intent to train an intent model. Collecting a substantial coverage of training data is a bottleneck in the bot building process. In cases where past conversation data is available, the cost of labeling hundreds of utterances with intent labels is time-consuming and laborious. In this paper, we present an intent discovery framework that can mine a vast amount of conversational logs and to generate labeled data sets for training intent models. We have introduced an extension to the DBSCAN (Ester et al., 1996) algorithm and presented a density-based clustering algorithm ITER-DBSCAN for unbalanced data clustering. Empirical evaluation on one conversation dataset, six intent dataset, and one short text clustering dataset show the effectiveness of our hypothesis. We release the datasets and code for future evaluation at https://github.com/ajaychatterjee/IntentMining . 1

show abstract

Section: Related Workmentioning

confidence: 99%

Intent Mining from past conversations for Conversational Agent

Chatterjee¹,

Sengupta²

2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…However, this model only used the textual information and not the localization and temporal information available from the tweets. Similarly, in 2019 Ghaemi and Farnaghi [42] proposed, VCDT (Varied Density-based spatial Clustering for Twitter data), an improvement on DBSCAN by incorporating spatial heterogeneity by identifying clusters of varying spatial densities. They considered the textual as well as spatial features of the Twitter data to detect the spatially clustered events.…”

Section: Clustering Based Approachesmentioning

confidence: 99%

An Approximate Model for Event Detection From Twitter Data

Dhiman

Toshniwal

2020

IEEE Access

View full text Add to dashboard Cite

The abundance and real-time availability of Twitter data have proved beneficial in detecting events in various domains such as emergency situations, crime detection, public health, place recommendations, etc. Nevertheless, two critical challenges occur while detecting events using social media data. First, the uncertainty in capturing the contextual relationship among tweets, which is the result of the limited availability of the contextual information due to the small length of tweets. Second, the high computation cost required in event detection due to massive data processing. Earlier research works, addressing these challenges, have tried to capture the contextual information by using the dense vector representations of texts leveraging deep neural word embedding generation models such as Word2Vec and GloVe. However, these models are trained on the Euclidean vector space which fails to amalgamate the directional information of the vectors with the semantic information in text, incurring high computational costs. To target both the problems simultaneously, we propose modeling Twitter data as a graph-of-sentences which retains the contextual relationships while maintaining lower computational cost. The proposed model captures contextual information using JoSE, a spherical vector representation leveraging the word-word and wordparagraph semantic co-occurrence statistics in a spherical generative model. Furthermore, the framework uses the weighted-graph model to capture all the relationships among the Twitter data efficiently. The graph is further pruned with the help of the graph component filtering approach. The graph clustering model, employed to detect the events, leverages the edge weights and the partial-k clustering approach maintaining low computation costs. The experimentation on the annotated benchmark Twitter data set and the real-world datasets show improved run-time performance up to 30% while maintaining the qualitative performance (F1-score) comparable to the state-of-the-art models.

show abstract

“…However, in actual line loss work, unqualified data may be produced in the process of collecting, transferring, and storing data. To address the issue, we applied the DBSCAN algorithm to cluster the data, which can realize anomaly detection [40,41]. DBSCAN is a clustering algorithm that, given a set of points in some space, groups together points that are closely packed together, marking points that lie alone in low-density regions as outliers.…”

Section: Distribution Network Classification Based On Dbscanmentioning

confidence: 99%

Research on Predicting Line Loss Rate in Low Voltage Distribution Network Based on Gradient Boosting Decision Tree

Yao

Zhu

et al. 2019

Energies

View full text Add to dashboard Cite

Line loss rate plays an essential role in evaluating the economic operation of power systems. However, in a low voltage (LV) distribution network, calculating line loss rate has become more cumbersome due to poor configuration of the measuring and detecting device, the difficulty in collecting operational data, and the excessive number of components and nodes. Most previous studies mainly focused on the approaches to calculate or predict line loss rate, but rarely involve the evaluation of the prediction results. In this paper, we propose an approach based on a gradient boosting decision tree (GBDT), to predict line loss rate. GBDT inherits the advantages of both statistical models and AI approaches, and can identify the complex and nonlinear relationship while computing the relative importance among variables. An empirical study on a data set in a city demonstrates that our proposed approach performs well in predicting line loss rate, given a large number of unlabeled examples. Experiments and analysis also confirmed the effectiveness of our proposed approach in anomaly detection and practical project management.

show abstract

A Varied Density-based Clustering Approach for Event Detection from Heterogeneous Twitter Data

Cited by 27 publications

References 50 publications

Intent Mining from past conversations for Conversational Agent

Intent Mining from past conversations for Conversational Agent

An Approximate Model for Event Detection From Twitter Data

Research on Predicting Line Loss Rate in Low Voltage Distribution Network Based on Gradient Boosting Decision Tree

Contact Info

Product

Resources

About