A Cost-based Optimizer for Gradient Descent Optimization

Abstract

As the use of machine learning (ML) permeates into diverse application domains, there is an urgent need to support a declarative framework for ML. Ideally, a user will specify an ML task in a high-level and easy-to-use language and the framework will invoke the appropriate algorithms and system configurations to execute it. An important observation towards designing such a framework is that many ML tasks can be expressed as mathematical optimization problems, which take a specific form. Furthermore, these optimization problems can be efficiently solved using variations of the gradient descent (GD) algorithm. Thus, to decouple a user specification of an ML task from its execution, a key component is a GD optimizer. We propose a cost-based GD optimizer that selects the best GD plan for a given ML task. To build our optimizer, we introduce a set of abstract operators for expressing GD algorithms and propose a novel approach to estimate the number of iterations a GD algorithm requires to converge. Extensive experiments on real and synthetic datasets show that our optimizer not only chooses the best GD plan but also allows for optimizations that achieve orders of magnitude performance speed-up.

RHEEM: enabling cross-platform data processing

“…However, Postgres is not as good as Spark for general purpose batch processing where parallel full scans are the key performance factor. Several studies have shown this kind of performance differences [20,34,40,53,61]. Diversity as Common Ground.…”

“…Moreover, today's data analytics is moving beyond the limits of a single platform. For example: (i) IBM reported that North York hospital needs to process 50 diverse datasets, which run on a dozen different platforms [38]; (ii) Airlines need to analyze large datasets, which are produced by different departments, are of different data formats, and reside on multiple data sources, to produce global reports for decision makers [9]; (iii) Oil & Gas companies need to process large amounts of diverse data spanning various platforms [19,36]; (iv) Several data warehouse applications require data to be moved from a MapReduce-like system into a DBMS for further analysis [28,56]; (v) Business intelligence typically requires an analytic pipeline composed of different platforms [58]; and (vi) Using multiple platforms for machine learning improves performance significantly [20,40]. Status Quo.…”

“…In the former case, we aim at seamlessly integrating all data analytic activity governing an aircraft; In the latter case, we aim at reducing the effort scientists need for building data analytic pipelines while at the same time speeding up the running time. Note that several papers show different aspects of Rheem: the vision behind it [17]; its optimizer [43]; its inequality join algorithm [42]; and a couple of its applications [40,41]. A couple of demo papers showcase the benefits of Rheem [16] and its interface [47].…”

“…In contrast to existing systems [29,30,34,58,62], Rheem helps users in all above cases. The design of our system has been mainly driven by four applications: a data cleaning application, BigDansing [41]; a machine learning application, ML4all [40]; a database application, xDB; and an end-toend data discovery and preparation application, Data Civilizer [32]. We use these applications to showcase the benefits of performing cross-platform data processing, instead of single-platform data processing, in terms of both performance and ease of use.…”

Downscaling Precipitation in the Data-Scarce Inland River Basin of Northwest China Based on Earth System Data Products

“…The least-squares method is sensitive to noise and suitable for relatively small samples [30,31]. Gradient descent algorithms are often used as the core methods of training algorithms in the field of machine learning, and they is commonly used to recursively approximate a minimum deviation model, such as regression and artificial neural networks [32][33][34][35][36][37]. The batch gradient descent (BGD) algorithm is a conventional method of gradient descent that is widely used in the field of machine learning [35,[38][39][40].…”