Second order design of multiclass kernel machines

Hao, Yilong; Tyagi, Kanishka; Rawat, Rohit; Manry, M.T.

doi:10.1109/ijcnn.2016.7727612

Cited by 2 publications

(1 citation statement)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The high dependency on first order gradient based training as in (D1) is an appealing area of research. Certainly, there is a need to investigate non-gradient methods like Grahm-Schmidt procedure along with our successful implementation of second order MLP training algorithms [83], [100], [122], [64], [47], [92]. As mentioned in (D2), the choice of activation function is extremely important while training the deep learner.…”

Section: Problems With Deep Learning Networkmentioning

confidence: 99%

Second Order Training and Sizing for the Multilayer Perceptron

Tyagi

Nguyen

Rawat

et al. 2019

Neural Process Lett

View full text Add to dashboard Cite

Training algorithms for deep learning have recently been proposed with notable success, beating the state-of-the-art in certain areas like audio, speech and language processing. The key role is played by learning multiple levels of abstractions in a deep architecture. However, searching the parameters space in a deep architecture is a difficult task. By exploiting the greedy layer-wise unsupervised training strategy of deep architecture, the network parameters are initialized near a good local minima. However, many existing deep learning algorithms require tuning a number of hyperparameters including learning factors and the number of hidden units in each layer. Apart from this, a predominant methodology in training deep learning models promotes the use of gradient-based algorithms which require heavy computational resources. Poor training algorithms and excessive user chosen parameters in a learning model makes it difficult to train a deep learner. In this dissertation, we break down the training of deep learning into basic building blocks of unsupervised approximation training followed by a supervised classification learning block. We propose a multi-step training method for designing generalized linear classifiers. First, an initial multi-class linear classifier is found through regression. Then validation error is minimized by pruning of unnecessary inputs. Simultaneously, desired outputs are improved via a method similar to the Ho-Kashyap rule. Next, the output discriminants are scaled to be net functions of sigmoidal output units in a generalized linear classifier. This classifier is trained via Newton's algorithm. Performance gains are demonstrated at each step. We then develop a family of batch training algorithm for the multi layer perceptron that optimizes its hidden layer size and number of training epochs. At the end of each training epoch, median filtering removes any kind of noise in the validation error vs number of hidden units curve and the networks get temporarily pruned. Since, pruning is done at each epoch, we save the best network thereby optimizing the number of hidden units as well as the number of epochs simultaneously. Next, we combine pruning with a growing approach. Later, the input units are scaled to be the net function of the sigmoidal output units that are then feed into as input to the MLP. We then propose resulting improvements in each of the deep learning blocks thereby improving the overall performance of the deep architecture. We discuss the principles and formulation regarding learning algorithms for deep autoencoders. We investigate several problems in deep autoencoders networks including training issues, the theoretical, mathematical and experimental justification that the networks are linear, optimizing the number of hidden units in each layer and determining the depth of the deep learning model. A direct implication of the current work is the ability to construct fast deep learning models using desktop level computational resources. This, in our opinion, promotes our desig...

show abstract

Section: Problems With Deep Learning Networkmentioning

confidence: 99%

Second Order Training and Sizing for the Multilayer Perceptron

Tyagi

Nguyen

Rawat

et al. 2019

Neural Process Lett

View full text Add to dashboard Cite

show abstract

Evaluating the Economic and Environmental Impacts of Comprehensive Emission Reduction Strategies in SMEs: A Case Study Approach

Patwa

2024

E3S Web Conf.

View full text Add to dashboard Cite

This paper assesses the economic and environmental implications of comprehensive reductions in emissions in small and medium enterprises. The document focuses on reduction from all three scopes, which are analyzed through a simple cost-benefit framework. The case studies of three companies in different sectors: automotive manufacturing, Textile, and Food processing, demonstrate substantial reduction of greenhouse gases, including 25%, 30%, and 12% reduction in scope 1, scope 2, and scope 3, respectively. Economical results feature reduced energy bills, tax deductions, and more efficient supply chains. Environmental implications include improved air quality, reduced waste, and lower carbon footprint. The policy implication is that similar strategies are beneficial for all businesses and offer attractive economic and operational opportunities. Furthermore, the paper can also help policymakers make data-driven decisions regarding the industry and the company. The method for future research comprises long-term economic and environmental effects, specific sectoral difficulties, and available reduction technologies.

show abstract

Second order design of multiclass kernel machines

Cited by 2 publications

References 26 publications

Second Order Training and Sizing for the Multilayer Perceptron

Second Order Training and Sizing for the Multilayer Perceptron

Evaluating the Economic and Environmental Impacts of Comprehensive Emission Reduction Strategies in SMEs: A Case Study Approach

Contact Info

Product

Resources

About