Deep Learning (DL) is a surprisingly successful branch of machine learning. The success of DL is usually explained by focusing analysis on a particular recent algorithm and its traits. Instead, we propose that an explanation of the success of DL must look at the population of all algorithms in the field and how they have evolved over time. We argue that cultural evolution is a useful framework to explain the success of DL. In analogy to biology, we use 'development' to mean the process converting the pseudocode or text description of an algorithm into a fully trained model. This includes writing the programming code, compiling and running the program, and training the model. If all parts of the process don't align well then the resultant model will be useless (if the code runs at all!). This is a constraint. A core component of evolutionary developmental biology is the concept of deconstraintsthese are modification to the developmental process that avoid complete failure by automatically accommodating changes in other components. We suggest that many important innovations in DL, from neural networks themselves to hyperparameter optimization and AutoGrad, can be seen as developmental deconstraints. These deconstraints can be very helpful to both the particular algorithm in how it handles challenges in implementation and the overall field of DL in how easy it is for new ideas to be generated. We highlight how our perspective can both advance DL and lead to new insights for evolutionary biology.
Evolution of deep learningDeep learning (DL) allows computers to learn enough from data to recognize faces (Schroff et al., 2015), play go (Schrittwieser et al., 2020), and model text (Brown et al., 2020) successfully. Coffee allows scientists to stay awake long enough to write another paper on what makes DL algorithms successful. How is it possible that caffeine has just the right structure to keep us awake enough to write this paper? Caffeine is an arrangement of 24 atoms in just the right way to resemble adenosine enough to occupy its receptors in brain cells but sufficiently different that it does not cause sleepiness (Huang et al., 2005). From the molecular perspective, this coincidence seems miraculous. Out of all the possible molecules made out of 24 atoms, the probability that a random one interacts with our neuroreceptors in just the right way is unbelievably small. This makes the proximate explanation -in terms that reduce to the interactions between atoms and molecules -for the success of caffeine feel unsatisfying relative to the ultimate explanation in terms of history.To be fully satisfied by an account of caffeine, we need to look at the history of this unlikely molecule. This history was both a process of cultural evolution -consuming herbs that affect us -and before that, a process of biological evolution -the coffee plant evolving a defense mechanism against insects that share some of their brain chemistry with us (Denoeud et al., 2014). Caffeine shares similar Preprint. Under review.