BMS: An improved Dunn index for Document Clustering validation

Misuraca, Michelangelo; Spano, Maria; Balbi, Simona

doi:10.1080/03610926.2018.1504968

Cited by 20 publications

(8 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DI [32] was used to validate and identify sets of clusters which were compact, with small variations between cluster members and with sufficient distance between other clusters centroids. The optimal number of clusters K=5 was chosen for this research work with maximum Dunn Index value (0.2797) as shown in Figure 5.…”

Section: And Discussionmentioning

confidence: 99%

“…These two phases are iterated for k=√ 𝑛 2 times, where n is the total number of feature vectors. In each iteration the Dunn index (DI) [32] is calculated using the equation 1 and the details of clusters and the DI values are stored in temporary lists. The total number of clusters and their details are considered for further analysis based on the highest DI value.…”

Section: Enhanced K-means Clusteringmentioning

confidence: 99%

See 1 more Smart Citation

Root cause analysis of COVID-19 cases by enhanced text mining process

Kokatnoor

Balachandran

2022

IJECE

View full text Add to dashboard Cite

<p>The main focus of this research is to find the reasons behind the fresh cases of COVID-19 from the public’s perception for data specific to India. The analysis is done using machine learning approaches and validating the inferences with medical professionals. The data processing and analysis is accomplished in three steps. First, the dimensionality of the vector space model (VSM) is reduced with improvised feature engineering (FE) process by using a weighted term frequency-inverse document frequency (TF-IDF) and forward scan trigrams (FST) followed by removal of weak features using feature hashing technique. In the second step, an enhanced K-means clustering algorithm is used for grouping, based on the public posts from Twitter®. In the last step, latent dirichlet allocation (LDA) is applied for discovering the trigram topics relevant to the reasons behind the increase of fresh COVID-19 cases. The enhanced K-means clustering improved Dunn index value by 18.11% when compared with the traditional K-means method. By incorporating improvised two-step FE process, LDA model improved by 14% in terms of coherence score and by 19% and 15% when compared with latent semantic analysis (LSA) and hierarchical dirichlet process (HDP) respectively thereby resulting in 14 root causes for spike in the disease.</p>

show abstract

Section: And Discussionmentioning

confidence: 99%

Section: Enhanced K-means Clusteringmentioning

confidence: 99%

Root cause analysis of COVID-19 cases by enhanced text mining process

Kokatnoor

Balachandran

2022

IJECE

View full text Add to dashboard Cite

show abstract

“…We experimented with 8 similarity thresholds from 0.6 to 0.95 with 0.05 increments to cluster distress narratives. Though various cluster quality metrics such as the Silhouette coefficient (Rousseeuw 1987), Dunn index (Misuraca, Spano, and Balbi 2019), and average point-to-centroid cosine distance, were computed for each threshold to select an optimal similarity threshold, manual inspection on a subset of 10 clusters at each threshold and cluster visualization revealed that those metrics do not work best for this dataset (Above metrics are known to work best only for datasets having convex-shaped clusters). Results of manual inspection conveyed that the stressors identified at higher thresholds such as 0.95 and 0.9 are too specific and those below 0.8 are too vague (cluster quality metrics and topics discovered through manual inspection at each threshold are included in the appendices).…”

Section: Identification Of Stressorsmentioning

confidence: 99%

HEAL: A Knowledge Graph for Distress Management Conversations

Welivita

2022

AAAI

View full text Add to dashboard Cite

The demands of the modern world are increasingly responsible for causing psychological burdens and bringing adverse impacts on our mental health. As a result, neural conversational agents with empathetic responding and distress management capabilities have recently gained popularity. However, existing end-to-end empathetic conversational agents often generate generic and repetitive empathetic statements such as "I am sorry to hear that", which fail to convey specificity to a given situation. Due to the lack of controllability in such models, they also impose the risk of generating toxic responses. Chatbots leveraging reasoning over knowledge graphs is seen as an efficient and fail-safe solution over end-to-end models. However, such resources are limited in the context of emotional distress. To address this, we introduce HEAL, a knowledge graph developed based on 1M distress narratives and their corresponding consoling responses curated from Reddit. It consists of 22K nodes identifying different types of stressors, speaker expectations, responses, and feedback types associated with distress dialogues and forms 104K connections between different types of nodes. Each node is associated with one of 41 affective states. Statistical and visual analysis conducted on HEAL reveals emotional dynamics between speakers and listeners in distress-oriented conversations and identifies useful response patterns leading to emotional relief. Automatic and human evaluation experiments show that HEAL's responses are more diverse, empathetic, and reliable compared to the baselines.

show abstract

“…The typical validity indices cannot solve the above problems. For example, the improved Dunn index [41] relies on the trail-and-error strategy and specific clustering algorithms. As for SCUBI, the intra-and inter-cluster distances cannot be defined well.…”

Section: Tibshirani's Gap Statistic Index (Gs ↑) [31]mentioning

confidence: 99%

Volume and Surface Area-Based Cluster Validity Index

Yue

Ding

2020

IEEE Access

View full text Add to dashboard Cite

Cluster validity index plays an important role in assessing the quality of clustering results. However, most of the existing validity indices take a trial-and-error strategy, and their correctness depend on not only the measurements of intra-and inter-cluster distances but also the specific clustering algorithms and data structures. Consequently, the applications of these indices are limited in practice. In this paper, we firstly define the total surface area and volume of all clusters in a 2-dimensinal data space, thereby recovering their natural interrelation among various numbers of clusters. On this basis, a novel validity index is proposed to directly assess the clustering results of any dataset, which does not require any trail-and-error process, clustering algorithms, data structures, or the measurements of intra-and inter-cluster distances. In the case of a high-dimensional data space, all clusters are transformed into spherical clusters of normalized size in a 2-dimensinal data space through a multidimensional scaling transformation. Two groups of typical synthetic datasets and real datasets with various characteristics are used to validate the novel validity index.INDEX TERMS Cluster validity index, multidimensional scaling transformation, volume and surface area.

show abstract

BMS: An improved Dunn index for Document Clustering validation

Cited by 20 publications

References 14 publications

Root cause analysis of COVID-19 cases by enhanced text mining process

Root cause analysis of COVID-19 cases by enhanced text mining process

HEAL: A Knowledge Graph for Distress Management Conversations

Volume and Surface Area-Based Cluster Validity Index

Contact Info

Product

Resources

About