Machine learning: statistical physics based theory and smart industry applications

Abstract

The contributions in this thesis are divided into two main parts: 1) a theoretical analysis of learning in neural networks and Learning Vector Quantization (LVQ) in model situations using statistical physics techniques and 2) the application of machine learning to smart industry settings.In the first part we address highly relevant situations and questions for current machine learning practice: using tools from statistical physics we analyse the learning behaviour in Rectified Linear Unit (ReLU) neural networks and compare it to sigmoidal neural networks in both on-line and off-line supervised learning settings, in order to contribute to the much needed theoretical insight into the properties of the use of the ReLU function, the most popular type of activation function in deep neural networks that are used in many machine learning tasks. Secondly, we analyse neural networks and LVQ under real and virtual concept drift processes that affect many applications of machine learning systems. Our analyses reveal several significant effects, which are, among others: ReLU networks handle overparameterization differently than sigmoidal networks, ReLU networks exhibit favourable second order phase transitions towards hidden unit specialization instead of the first order phase transitions observed for sigmoidal networks and applying weight decay in concept drift scenarios is more effective for ReLU neural networks, in which it significantly accelerates the onset of specialization. For LVQ we find non-trivial dependence of the generalization performance on the learning rate in concept drift situations. Moreover, it is shown that an appropriate amount of weight decay can be beneficial to the performance in the real drift settings. In contrast, the resulting limited flexibility of the prototypes decreases the performance under virtual drift.In the second part of the thesis we focus on the use of computational intelligence approaches in applications in industry. First, we perform a typical Industry 4.0 case study in collaboration with industry that concerns the development of real-time material quality control in a highthroughput production line of steel-based products. In this case study, material measurements iii taken with a fast non-invasive sensor are related to material properties measured by tensile tests. Due to significant correlations between the two types of measurements, we successfully fit and evaluate a model that can estimate material properties in real-time. Furthermore, it is shown on 108 kilometres of processed steel that the model is able to prevent expensive production problems and that it can indicate a risk of the occurrence of production faults.Additionally, we propose a methodology for time series classification that combines Generalized Matrix Learning Vector Quantization (GMLVQ) formulated for complex-valued data with complex-valued Fourier and wavelet features. On several benchmark datasets and a heart beat classification task, learning in the space of complex-valued coefficients is found to yield bett...