Evaluation of a machine learning classifier for metamodels

Abstract

Modeling is a ubiquitous activity in the process of software development. In recent years, such an activity has reached a high degree of intricacy, guided by the heterogeneity of the components, data sources, and tasks. The democratized use of models has led to the necessity for suitable machinery for mining modeling repositories. Among others, the classification of metamodels into independent categories facilitates personalized searches by boosting the visibility of metamodels. Nevertheless, the manual classification of metamodels is not only a tedious but also an error-prone task. According to our observation, misclassification is the norm which leads to a reduction in reachability as well as reusability of metamodels. Handling such complexity requires suitable tooling to leverage raw data into practical knowledge that can help modelers with their daily tasks. In our previous work, we proposed AURORA as a machine learning classifier for metamodel repositories. In this paper, we present a thorough evaluation of the system by taking into consideration different settings as well as evaluation metrics. More importantly, we improve the original AURORA tool by changing its internal design. Experimental results demonstrate that the proposed amendment is beneficial to the classification of metamodels. We also compared our approach with two baseline algorithms, namely gradient boosted decision tree and support vector machines. Eventually, we see that AURORA outperforms the baselines with respect to various quality metrics.

ModelXGlue: a benchmarking framework for ML tools in MDE

“…Model classification. AURORA [36,37] is a tool that exploits a feed-forward neural network to classify metamodels. The authors proved the tool's capability to classify Ecore models accurately.…”

“…To support such a statement and introduce the challenges involved in benchmarking ML/MDE tasks, we discuss the model classification task, which has been subject to different proposals over the last few years [32,37]. The task consists of assigning a meaningful label to a given model based on the labels observed in the training data.…”