AutoElbow: An Automatic Elbow Detection Method for Estimating the Number of Clusters in a Dataset

Onumanyi, A. J.; Molokomme, Daisy Nkele; Isaac, Sherrin J.; Abu‐Mahfouz, Adnan M.

doi:10.3390/app12157515

Cited by 27 publications

(10 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To provide spatially variable coastal dynamics characteristics, we used the K-means algorithm 37 to cluster the HTF threshold’s input data into different categories. The elbow approach 38 is used to determine the optimal number of clusters. This approach plots the number of clusters against the total distance between data points and their respective cluster centroids and identifies the elbow point on the plot, which indicates a trade-off between minimizing the distance and mitigating the complexity associated with an excessive number of clusters.…”

Section: Resultsmentioning

confidence: 99%

Establishing flood thresholds for sea level rise impact communication

Mahmoudi,

Moftakhari,

Muñoz

et al. 2024

Nat Commun

View full text Add to dashboard Cite

Sea level rise (SLR) affects coastal flood regimes and poses serious challenges to flood risk management, particularly on ungauged coasts. To address the challenge of monitoring SLR at local scales, we propose a high tide flood (HTF) thresholding system that leverages machine learning (ML) techniques to estimate SLR and HTF thresholds at a relatively fine spatial resolution (10 km) along the United States’ coastlines. The proposed system, complementing conventional linear- and point-based estimations of HTF thresholds and SLR rates, can estimate these values at ungauged stretches of the coast. Trained and validated against National Oceanic and Atmospheric Administration (NOAA) gauge data, our system demonstrates promising skills with an average Kling-Gupta Efficiency (KGE) of 0.77. The results can raise community awareness about SLR impacts by documenting the chronic signal of HTF and providing useful information for adaptation planning. The findings encourage further application of ML in achieving spatially distributed thresholds.

show abstract

Section: Resultsmentioning

confidence: 99%

Establishing flood thresholds for sea level rise impact communication

Mahmoudi,

Moftakhari,

Muñoz

et al. 2024

Nat Commun

View full text Add to dashboard Cite

show abstract

“…L-method places two straight lines on the elbow curve. One line extends from the head of the curve to the candidate point on the curve, while the other line extends from the tail of the curve to the candidate point on the same curve [77]. Using this technique, we first compute the Bayesian Information Criterion (BIC) values, as well as the first-order differences.…”

Section: L-methods Techniquementioning

confidence: 99%

“…However, with the L-method, long tails within the curve can influence the value of the optimal number of clusters, k [77]. To minimize this effect, a proposed iterative method aims to gradually decrease the tail while simultaneously refining the point of the elbow.…”

Section: L-methods Techniquementioning

confidence: 99%

See 1 more Smart Citation

K-Hyperparameter Tuning in High-Dimensional Space Clustering: Solving Smooth Elbow Challenges Using an Ensemble Based Technique of a Self-Adapting Autoencoder and Internal Validation Indexes

Gikera,

Mwaura,

Muuro

et al. 2023

Journal on Artificial Intelligence

View full text Add to dashboard Cite

k-means is a popular clustering algorithm because of its simplicity and scalability to handle large datasets. However, one of its setbacks is the challenge of identifying the correct k-hyperparameter value. Tuning this value correctly is critical for building effective k-means models. The use of the traditional elbow method to help identify this value has a long-standing literature. However, when using this method with certain datasets, smooth curves may appear, making it challenging to identify the k-value due to its unclear nature. On the other hand, various internal validation indexes, which are proposed as a solution to this issue, may be inconsistent. Although various techniques for solving smooth elbow challenges exist, k-hyperparameter tuning in high-dimensional spaces still remains intractable and an open research issue. In this paper, we have first reviewed the existing techniques for solving smooth elbow challenges. The identified research gaps are then utilized in the development of the new technique. The new technique, referred to as the ensemble-based technique of a self-adapting autoencoder and internal validation indexes, is then validated in high-dimensional space clustering. The optimal k-value, tuned by this technique using a voting scheme, is a trade-off between the number of clusters visualized in the autoencoder's latent space, k-value from the ensemble internal validation index score and one that generates a value of 0 or close to 0 on the derivativeat the elbow. Experimental results based on the Cochran's Q test, ANOVA, and McNemar's score indicate a relatively good performance of the newly developed technique in k-hyperparameter tuning. KEYWORDS k-hyperparameter tuning; high-dimensional; smooth elbow K-Means ArchitectureK-means is an iterative algorithm that aims to partition a dataset into a set of k non-overlapping groups of data points [9]. The k-hyperparameter is one of the most important hyperparameters to tune in k-means [10,11]. Tuning a machine learning model's hyperparameters has a significant effect on its performance [12]. In this subsection, we explore k-clusters, k-hyperparameters, and the traditional elbow method used to identify the optimal value for the k-hyperparameter. K-Means ClustersThe k-means clusters are the resulting data sub-groups generated by the popular unsupervised partitioning algorithm, known as the k-means clustering algorithm [13]. Cluster analysis using the k-means algorithm is an example of a k-means-based model that has been successfully applied in cluster analysis [14,15]. The k-clusters generated by these algorithms are usually composed of distinct non-overlapping groups of data points, aggregated together because they share specific similarities [16]. The data points within a particular cluster are similar, while the data points across different clusters are different [17]. Both the intra-cluster and inter-cluster data points are measured using a sum of squared distance-based metric [18,19]. For this reason, the original un-partitioned dataset is stan...

show abstract

“…By observing the WCSS curve, we looked for a point where the rate of decrease in WCSS significantly slowed down, creating an elbow in the plot. The K value at this elbow point is considered the optimal number of clusters, as it indicates a trade-off between maximizing the number of clusters and minimizing WCSS [53].…”

mentioning

confidence: 99%

A Novel Statistical Framework for Optimal Sizing of Grid-Connected Photovoltaic–Battery Systems for Peak Demand Reduction to Flatten Daily Load Profiles

Nematirad,

Pahwa,

Natarajan

2024

Solar

View full text Add to dashboard Cite

Integrating photovoltaic (PV) systems plays a pivotal role in the global shift toward renewable energy, offering significant environmental benefits. However, the PV installation should provide financial benefits for the utilities. Considering that the utility companies often incur costs for both energy and peak demand, PV installations should aim to reduce both energy and peak demand charges. Although PV systems can reduce energy needs during the day, their effectiveness in reducing peak demand, particularly in the early morning and late evening, is limited, as PV generation is zero or negligible at those times. To address this limitation, battery storage systems are utilized for storing energy during off-peak hours and releasing it during peak times. However, finding the optimal size of PV and the accompanying battery remains a challenge. While valuable optimization models have been developed to determine the optimal size of PV–battery systems, a certain gap remains where peak demand reduction has not been sufficiently addressed in the optimization process. Recognizing this gap, this study proposes a novel statistical model to optimize PV–battery system size for peak demand reduction. The model aims to flatten 95% of daily peak demands up to a certain demand threshold, ensuring consistent energy supply and financial benefit for utility companies. A straightforward and effective search methodology is employed to determine the optimal system sizes. Additionally, the model’s effectiveness is rigorously tested through a modified Monte Carlo simulation coupled with time series clustering to generate various scenarios to assess performance under different conditions. The results indicate that the optimal PV–battery system successfully flattens 95% of daily peak demand with a selected threshold of 2000 kW, yielding a financial benefit of USD 812,648 over 20 years.

show abstract

AutoElbow: An Automatic Elbow Detection Method for Estimating the Number of Clusters in a Dataset

Cited by 27 publications

References 17 publications

Establishing flood thresholds for sea level rise impact communication

Establishing flood thresholds for sea level rise impact communication

K-Hyperparameter Tuning in High-Dimensional Space Clustering: Solving Smooth Elbow Challenges Using an Ensemble Based Technique of a Self-Adapting Autoencoder and Internal Validation Indexes

A Novel Statistical Framework for Optimal Sizing of Grid-Connected Photovoltaic–Battery Systems for Peak Demand Reduction to Flatten Daily Load Profiles

Contact Info

Product

Resources

About