The computational discovery of novel materials has been one of the main motivations behind research in theoretical chemistry for several decades. Despite much effort, this is far from a solved problem, however. Among other reasons, this is due to the enormous space of possible structures and compositions that could potentially be of interest. In the case of inorganic materials, this is exacerbated by the combinatorics of the periodic table, since even a single crystal structure can in principle display millions of compositions. Consequently, there is a need for tools that enable a more guided exploration of the materials design space. Here, generative machine learning (ML) models have recently emerged as a promising technology. In this work, we assess the performance of a range of deep generative models based on Reinforcement Learning (RL), Variational Autoencoders (VAE) and Generative Adversarial Networks (GAN) for the prototypical case of designing Elpasolite compositions with low formation energies. By relying on the fully enumerated space of 2~million main group Elpasolites, the precision, coverage and diversity of the generated materials is rigorously assessed. Additionally, a hyperparameter selection scheme for generative models in chemical composition space is developed.