Economic deterministic ensemble classifiers with probabilistic output using for robust quantification: study of unbalanced educational datasets

Abstract

The overall goal of our work is to find economic and robust supervised machine learning methods which adequate to both individual and collective Student Performance Forecast (SPF). The individual SPF are subject of well-known classification methods but collective SPF is subject of quantification learning algorithms dealing with the novel task to predict the frequency of classes in tested sample e.g. a number of students with unsatisfactory grade. The need for revise of classification methods shows review of 86 SPF in developing countries. The analysis depicts that most of SPF report the high overall accuracy of classifiers based on decision tree J48, Naïve Base NB, Multilayer Perception MLP, k-Nearest Neighbor k-NN, and Support Vector Machine SVM algorithms, but did not take into account the accuracy of the forecast of a minor presented class. So, given the imbalance in the sample, "useful forecast" with the F1 metric above 50% (75%) are given only in ½ (1/5) of cases of forecasts. The pivotal study of the efficacy factors of binary SPF (data type, algorithm, sample balancing, number of classes etc.). Another important finding is that classifiers with the probabilistic Naïve Bayesian kernel, have more stable behavior to classify different EDM datasets, overcoming MLP, J48, SVM and k-NN based classifiers which sometimes achieved good forecast but sometimes failed in prediction. After that, collected all the above experimental finds associated with relationship between algorithm and data information, we construct 3-15 member heterogeneous ensembles contained strong, moderate and weak classifiers for deterministic individual SPF by simple voting and heuristically proposed how individual probabilistic predictions could be generated and how to aggregate them for overall frequency forecasting, i.e. resolve the task of quantification. The proposed methods of ensemble forecasting and ensemble quantification can become the basis for new economic and robust solutions of various real-world problems in the field of machine learning.

Monitoring of the Education Quality and Implementing of Individual Learning: Demonstration of Approaches and Educational Data Mining Algorithms

“…Particularly, we have demonstrated that decentralized national education systems with school-based management do not automatically lead to a rise of education quality. We conclude that at the local and regional level MQE, a key part of system is the prediction of a new situation with the quality of learning, determined by the individual and collective achievements of students, which in turn are educational data mining problems resolving by the classification and quantification methods [6][7]. Obviously, the mass-line courses implement [8] and recent ubiquitous implementation of distance learning as consequence of COVID-19 pandemic, also requires the application of these methods.…”

“…It is usually said that these datasets are balanced. However, detailed study of the student performance forecasts [7] showed that reviewed above high values of A were most often caused by imbalance of educational data samples. Indeed, our entire education systems are set up in such a way that the number of students N+ is much higher than N-Then, if the binary classifier mistakenly categorizes all new unknown students as a class of c+, the number of correct predictions in (2) will be equal to the N+, and the accuracy of this weak classifier A w will be equal to relative frequency p + :of dominant class c+:…”

“…On the other hand, the p+ vs. p-imbalance results that binary classifier will be trained on a larger amount N+ of data corresponding to features of dominant class, and, as consequence, on the strongly unbalanced samples p+>>p-, the minority class of academic performance is virtually ignored by forecasting procedure. These effects can be seen from Table 1 where are demonstrated the results of forecast of 10 different types of classifiers trained by using Weka 3.8 [10] on the samples of 100 students derived from 6 experimental unbalanced dataset which, as we are sure [7,[11][12][13][14][15], have reasonable pre-processing quality.…”

“…And as in real-world case, the technology for obtaining ensemble forecast results is quite simple and robust. Following the work [7], we will replace the labels "с+" and "с-" of binary classification by integers +1 and -1, and will consider the forecast of the binary classifier of model h t for vector x i . It is clear that which will take two values h t (x i )=y t ∊ {-1,1}.…”

“…Thus, the obvious advantage of the presented technology of heterogeneous ensemble in that it gives higher and more stable results of the forecast, allows easily to incorporate new ensemble member: any classifiers or other ensembles. It is obvious that you can consciously select sets of ensemble members, for example, conducting a preliminary ranking of the classifiers on the stability of forecasts [7].…”