Single-junction photovoltaic devices exhibit a bottleneck in their efficiency due to incomplete or inefficient harvesting of photons in the low- or high-energy regions of the solar spectrum. Spectral converters can be used to convert solar photons into energies that are more effectively captured by the photovoltaic device through a photoluminescence process. Here, recent advances in the fields of luminescent solar concentration, luminescent downshifting, and upconversion are discussed. The focus is specifically on the role that materials science has to play in overcoming barriers in the optical performance in all spectral converters and on their successful integration with both established (e.g., c-Si, GaAs) and emerging (perovskite, organic, dye-sensitized) cell types. Current challenges and emerging research directions, which need to be addressed for the development of next-generation luminescent solar devices, are also discussed.