Symbolic and automatic differentiation of languages

Abstract

Formal languages are usually defined in terms of set theory. Choosing type theory instead gives us languages as type-level predicates over strings. Applying a language to a string yields a type whose elements are language membership proofs describing how a string parses in the language. The usual building blocks of languages (including union, concatenation, and Kleene closure) have precise and compelling specifications uncomplicated by operational strategies and are easily generalized to a few general domain-transforming and codomain-transforming operations on predicates. A simple characterization of languages (and indeed functions from lists to any type) captures the essential idea behind language “differentiation” as used for recognizing languages, leading to a collection of lemmas about type-level predicates. These lemmas are the heart of two dual parsing implementations—using (inductive) regular expressions and (coinductive) tries—each containing the same code but in dual arrangements (with representation and primitive operations trading places). The regular expression version corresponds to symbolic differentiation, while the trie version corresponds to automatic differentiation. The relatively easy-to-prove properties of type-level languages transfer almost effortlessly to the decidable implementations. In particular, despite the inductive and coinductive nature of regular expressions and tries respectively, we need neither inductive nor coinductive/bisimulation arguments to prove algebraic properties.

Differentiable Imaging: A New Tool for Computational Optical Imaging

“…Implementing differentiable imaging can be challenging as it demands a combination of physical and mathematical expertise to properly formulate the problem, encode relevant priors mathematically, and determine which parts of the code need differentiation. [ 94 ] Additionally, for imaging applications that involve interactions between light and matter, such as biology or material science, domain‐specific knowledge may also be necessary. [ 16,17 ] Effective modeling requires a deep understanding of the differentiable programming language .…”

Forward- or Reverse-Mode Automatic Differentiation: What's the Difference?

“…Next, he shifts his view to derivatives as linear maps and the categorical structure of differentiation [31]. More recently, Elliott has also explored the application of automatic differentiation to languages [32].…”

Combinatory Adjoints and Differentiation

“…Functional languages have served well for exploring AD techniques both as a host language for capturing AD techniques [31,32], and as the language under investigation, featuring, for instance, multivariate functions, higher-dimensional data, higher-order functions, higher-degree differentiation (e.g., through a lazy infinite tower of derivatives) [15,22], and even differentiation of formal languages [14]. Whereas many of the above-mentioned features are well-suited for forward-mode AD (no memoization of primal values is needed), capturing the essence of reverse-mode AD has proven difficult.…”