Backprop as Functor: A compositional perspective on supervised learning

Abstract

A supervised learning algorithm searches over a set of functions A → B parametrised by a space P to find the best approximation to some ideal function f : A → B. It does this by taking examples (a, f (a)) ∈ A × B, and updating the parameter according to some rule. We define a category where these update rules may be composed, and show that gradient descent-with respect to a fixed step size and an error function satisfying a certain property-defines a monoidal functor from a category of parametrised functions to this category of update rules. A key contribution is the notion of request function. This provides a structural perspective on backpropagation, giving a broad generalisation of neural networks and linking it with structures from bidirectional programming and open games.

General Supervised Learning as Change Propagation with Delta Lenses

“…To manage the extended search possibilities, it makes sense to parameterize the space of transformations as a family of mappings get p : A → B indexed over some parameter space p ∈ P. For example, we may define the ITdepartments view to be parameterized by the experience of employees shown in the view (including any experience as a special parameter value). Then we have two interrelated propagation operations that map an update B B to a parameter update p p and a source update A A (called, resp., an update and a request in [1]). Thus, the extended search space allows for new update policies that look for updating the parameter as an update propagation possibility.…”

“…As model spaces are themselves categories, the entire search space is a product of categories, P×A (or even P×A×B if we consider amendments), and thus codiscreteness may "affect" only P or only A or both. For example, for learners described in [1], spaces A, B, P are sets, get is a parameterized function, and put consists of two families of discrete operations described in [1]: put upd updates the parameter and put req updates the source value thus making a request to the previous layer. In general, a learning lens from a model space (category) A to model space B is a pair of operations (get, put): A P -B where get: P×A → B is a functor considered (via Currying) as a family of functors get: P → B A , and put is a family of operations providing some sort of an inverse map for functor get.…”

“…The category of learning delta lenses with amendments (and its subcategory of lenses without amendment) are defined in the present paper (see Sections 4 and 5) and located at point (2,2). There are also two semicategorificated species of learning lenses: categorical learners at point (1,2) and codiscretely learning delta lenses at (2,1), which are special cases of ala-lenses. Fig.…”

“…In a seminal paper [1], Fong, Spivak and Tuyéras showed how to compose supervised machine learning (ML) algorithms so that the latter form a symmetric monoidal (sm) category Learn Learn Learn, and built an sm-functor (1) L ε,err : P ara P ara P ara R → Learn Learn Learn, which maps a parameterized differentiable function f : P × R m → R n (with P = R k being the parameter space ) to a learning algorithm called a learner; the latter improves an initially given function f (p, _): R m → R n by learning from a set of training pairs (a, b) ∈ R m ×R n . The functor is itself parameterized by a step size 0 < ε ∈ R and an error function err: R×R → R needed to specify the subject function of the gradient descent procedure.…”

“…The compositional nature of the gradient descent method in ML shown in [1], i.e., the existence of functor L ε,err specified in diagram (a), can be seen as a (codiscrete) instance of the construct specified by diagram (c) (see Example 4 on p. 19 for a more accurate formulation). Finding specifically Bx non-codiscrete instances is a future work as checking compositionality of a given model transformation language is a laborious endeavour.…”