A machine is taught by finding the minimum value of the cost function which is induced by learning data. Unfortunately, as the amount of learning increases, the non-liner activation function in the artificial neural network (ANN), the complexity of the artificial intelligence structures, and the cost function’s non-convex complexity all increase. We know that a non-convex function has local minimums, and that the first derivative of the cost function is zero at a local minimum. Therefore, the methods based on a gradient descent optimization do not undergo further change when they fall to a local minimum because they are based on the first derivative of the cost function. This paper introduces a novel optimization method to make machine learning more efficient. In other words, we construct an effective optimization method for non-convex cost function. The proposed method solves the problem of falling into a local minimum by adding the cost function in the parameter update rule of the ADAM method. We prove the convergence of the sequences generated from the proposed method and the superiority of the proposed method by numerical comparison with gradient descent (GD, ADAM, and AdaMax).

No abstract

The process of machine learning is to find parameters that minimize the cost function constructed by learning the data. This is called optimization and the parameters at that time are called the optimal parameters in neural networks. In the process of finding the optimization, there were attempts to solve the symmetric optimization or initialize the parameters symmetrically. Furthermore, in order to obtain the optimal parameters, the existing methods have used methods in which the learning rate is decreased over the iteration time or is changed according to a certain ratio. These methods are a monotonically decreasing method at a constant rate according to the iteration time. Our idea is to make the learning rate changeable unlike the monotonically decreasing method. We introduce a method to find the optimal parameters which adaptively changes the learning rate according to the value of the cost function. Therefore, when the cost function is optimized, the learning is complete and the optimal parameters are obtained. This paper proves that the method ensures convergence to the optimal parameters. This means that our method achieves a minimum of the cost function (or effective learning). Numerical experiments demonstrate that learning is good effective when using the proposed learning rate schedule in various situations.

This paper analyzes the operation principle and predicted value of the recurrent-neural-network (RNN) structure, which is the most basic and suitable for the change of time in the structure of a neural network for various types of artificial intelligence (AI). In particular, an RNN in which all connections are symmetric guarantees that it will converge. The operating principle of a RNN is based on linear data combinations and is composed through the synthesis of nonlinear activation functions. Linear combined data are similar to the autoregressive-moving average (ARMA) method of statistical processing. However, distortion due to the nonlinear activation function in RNNs causes the predicted value to be different from the predicted ARMA value. Through this, we know the limit of the predicted value of an RNN and the range of prediction that changes according to the learning data. In addition to mathematical proofs, numerical experiments confirmed our claims.

Abstract. In this paper, we develop an iteration free backward semi-Lagrangian method for nonlinear guiding center models. We apply the fourth-order central difference scheme for the Poisson equation and employ the local cubic interpolation for the spatial discretization. A key problem in the time discretization is to find the characteristic curve arriving at each grid point which is the solution of a system of highly nonlinear ODEs with a self-consistency imposed by the Poisson equation. The proposed method is based on the error correction method recently developed by the authors. For the error correction method, we introduce a modified Euler's polygon and solve the induced asymptotically linear differential equation with the midpoint quadrature rule to get the error correction term. We prove that the proposed iteration free method has convergence order at least 3 in space and 2 in time in the sense of the L 2 -norm. In particular, it is shown that the proposed method has a good performance in computational cost together with better conservation properties in mass, the total kinetic energy, and the enstrophy compared to the conventional second-order methods. Numerical test results are presented to support the theoretical analysis and discuss the properties of the newly proposed scheme. 1. Introduction. The model problem we are concerned with is the guiding center model, which was developed for an efficient description of low-frequency turbulence and resulting transport phenomena in strongly magnetized plasmas. Instead of tracing the fast gyro-motions of charged particles under strong external magnetic fields, the guiding center model follows the evolution of the center of the fast gyro-motions, which allows an efficient description of charged particle dynamics under relatively slow electrostatic fluctuations ω e ω. Here, ω denotes the gyro-frequency of a charged particle. If we suppose a uniform external magnetic field and the plane perpendicular to the magnetic field, the density of the guiding centers of charged particles, which are interacting with each other through self-consistent electrostatic potential, satisfies the following form of nonlinear hyperbolic equation in the plane with a proper normalization: