Cellular dynamics and emerging biological function are governed by patterns of gene expression arising from networks of interacting genes. Inferring these interactions from data is a notoriously difficult inverse problem that is central to systems biology. The majority of existing network inference methods work at the population level and construct a static representations of gene regulatory networks; they do not naturally allow for inference of differential regulation across a heterogeneous cell population. Building upon recent dynamical inference methods that model single cell dynamics using Markov processes, we propose locaTE, an information-theoretic approach which employs a localised transfer entropy to infer cell-specific, causal gene regulatory networks. LocaTE uses high-resolution estimates of dynamics and geometry of the cellular gene expression manifold to inform inference of regulatory interactions. We find that this approach is generally superior to using static inference methods, often by a significant margin. We demonstrate that factor analysis can give detailed insights into the inferred cell-specific GRNs. In application to two experimental datasets, we recover key transcription factors and regulatory interactions that drive mouse primitive endoderm formation and pancreatic development. For both simulated and experimental data, locaTE provides a powerful, efficient and scalable network inference method that allows us to distil cell-specific networks from single cell data.