Model Stability with Continuous Data Updates

Liu, Huiting; Avinesh, P. V. S.; Patwardhan, Siddharth V.; Grasch, Peter; Agarwal, Sachin

doi:10.48550/arxiv.2201.05692

Cited by 1 publication

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The disagreement is easily calculated in practice by training a number of models and then averaging the pairwise disagreements. This measure is also known as churn [1,2,6,12] and jitter [9].…”

Section: Preliminaries and Experimental Setupmentioning

confidence: 99%

“…Model Influence. Liu et al [9] study how data updates affect prediction stability in the domain of language processing. Moreover, they compare whether model architecture, model complexity, or usage of pretrained word embeddings improve stability.…”

Section: Related Workmentioning

confidence: 99%

“…overall system performance deteriorates unpredictably despite improvement of the retrained model [9]. Finally, the reproducibility of individual predictions is important in critical domains such as finance or medicine, in which recommendations reliant on (for example) random initializations might not be acceptable.…”

Section: Introductionmentioning

confidence: 99%

“…Finally, the reproducibility of individual predictions is important in critical domains such as finance or medicine, in which recommendations reliant on (for example) random initializations might not be acceptable. Due to this importance, there has been a recent surge of work studying the prediction instability of machine learning models [2,6,9,12,15,17,21]. However, research on the instability of models in graphs/network settings, such as node classification, has received little attention so far.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

On the Prediction Instability of Graph Neural Networks

Klabunde¹,

Lemmerich²

2022

Preprint

View full text Add to dashboard Cite

Instability of trained models, i.e., the dependence of individual node predictions on random factors, can affect reproducibility, reliability, and trust in machine learning systems. In this paper, we systematically assess the prediction instability of node classification with state-of-the-art Graph Neural Networks (GNNs). With our experiments, we establish that multiple instantiations of popular GNN models trained on the same data with the same model hyperparameters result in almost identical aggregated performance, but display substantial disagreement in the predictions for individual nodes. We find that up to one third of the incorrectly classified nodes differ across algorithm runs. We identify correlations between hyperparameters, node properties, and the size of the training set with the stability of predictions. In general, maximizing model performance implicitly also reduces model instability. 1

show abstract

Section: Preliminaries and Experimental Setupmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%