Gene regulatory network inference methods are often heuristically designed for specific datasets and biological problems, making it challenging to include extra information and compare against other algorithms. Here, we propose the use of probabilistic matrix factorization for gene regulatory network (PMF-GRN) inference from single-cell gene expression datasets of any size, incorporating experimental evidence into prior distributions over latent factors. We use variational inference to infer GRNs, enabling hyperparameter search and direct comparison to other generative models. We evaluate our method using Saccharomyces cerevisiae and Bacillus subtilis, benchmarking against database-derived gold standard GRNs. We find that PMF-GRN infers GRNs more accurately than the current state-of-the-art method, while reducing the need for heuristic model selection. We demonstrate the necessity of incorporating prior information into any matrix factorization approach to GRN inference. Finally, we find that PMF-GRN's uncertainty estimates are well-calibrated.