CHAD for Expressive Total Languages

Abstract

We show how to apply forward and reverse mode Combinatory Homomorphic Automatic Differentiation (CHAD) (Vákár, 2021 ,b) to total functional programming languages with expressive type systems featuring the combination of • tuple types; • sum types; • inductive types; • coinductive types; • function types.We achieve this by analysing the categorical semantics of such types in Σ-types (Grothendieck constructions) of suitable categories. Using a novel categorical logical relations technique for such expressive type systems, we give a correctness proof of CHAD in this setting by showing that it computes the usual mathematical derivative of the function that the original program implements. The result is a principled, purely functional and provably correct method for performing forward and reverse mode automatic differentiation (AD) on total functional programming languages with expressive type systems.

Logical Relations for Partial Features and Automatic Differentiation Correctness

“…In particular, we believe that it provides a simple, principled denotational explaination of how AD and expressive partial language features should interact. We plan to use it to generalise and prove correct the more advanced AD technique CHAD [34,35,19] when applied to languages with partial features.…”

“…There has recently been a flurry of work studying AD from a programming language point of view, a lot of it focussing on functional formulations of AD and their correctness. Examples of such papers are [24,9,27,6,1,13,21,34,19,14,35,16,29]. Of these papers, [24,1,21,29] are particularly relevant as they also consider automatic differentiation of languages with partial features.…”

“…For example, the authors have given such proofs in the past (in the preprint [33]), but despite our best efforts they seemed to require subtle combinations of sheaves (to deal with the surprisingly hard interaction between partiality and differentiation induced by conditionals on real numbers) and ω-cpo-structure (required for recursion). The resulting proofs were dissatisfying due to their complexity, restricting the audience and obstructing their generalisation to more advanced AD algorithms like CHAD [35,19].…”

“…The present work is a vast simplification of the arguments in [33] (and, also, [13]), not requiring any (ω-cpos internal to) sheaves or diffeological spaces and instead relying on plain ω-cpos. This simplification is desirable if we are to apply the techniques more generally, such as to the correctness of CHAD [35,19] for recursion, and if we want the technique to be more broadly accessible.…”