Binding of transcription factors (TFs) at proximal promoters and distal enhancers is central to gene regulation. Yet, identification of TF binding sites, also known as regulatory motifs, and quantification of their impact remains challenging. Here we present scover, a convolutional neural network model that can discover putative regulatory motifs along with their cell type-specific importance from single-cell data. Analysis of scRNA-seq data from human kidney shows that ETS, YY1 and NRF1 are the most important motif families for proximal promoters. Using multiple mouse tissues we obtain for the first time a model with cell type resolution which explains 34% of the variance in gene expression. Finally, by applying scover to distal enhancers identified using scATAC-seq from the mouse cerebral cortex we highlight the emergence of layer specific regulatory patterns during development.