Human infants have ‘core knowledge systems’ that support basic intuitions about the world including objects and their motion, space, number, and time. What is the origin of these systems, and what is their nature? Although often regarded as separate, domain-specific modules, evidence for similar abilities across many nonhuman species suggests that core systems might be integrated, consistent with views of modularity in evolutionary-developmental biology. Here we propose that core knowledge systems are based on an ability to form representations of the environment with algebraic structure – that is, on implicit computation. Algebraic groups encode symmetries, with computation inherent in the structure – a view that complements an understanding of computation as action or function. Our proposal is related to previous applications of group theory in perception and computational-representational accounts of learning (Gallistel, 1990), but suggests for the first time a common basis for core knowledge across humans and nonhumans. Implicit computation can be studied experimentally with an ‘artificial algebra’ task in which adults learn to respond based on arithmetic combinations of stimulus magnitudes, by feedback and without explicit instruction. Asking why organisms have a capacity for implicit computation suggests two possibilities: Either the geometric invariants of the world have been internalized in perceptual systems by natural selection (Shepard, 1994), or mathematical structure is intrinsic to the mind. Understood more broadly in a framework offered by Penrose (2004), implicit computation is a linchpin with potential to unlock some of the most fundamental questions about relationships between the mind, mathematics, and the world.