Silicon nitride compounds emit photoluminescence all over the visible range. Recent studies ascribed this luminescence to quantum-size effects within silicon nanocrystals that were either shown or assumed to form inside the silicon nitride matrix; the luminescence of the matrix itself was ignored. In contrast, observing the same luminescence even without the presence of silicon crystallites, our work identifies the silicon nitride matrix itself as responsible for the photoluminescence. All experimental observations are well explained by band tail luminescence from the silicon matrix. In contrast to the silicon nanocrystal approach, our model explains all aspects of the luminescence. As a consequence, we conclude that silicon nitride films are inappropriate if one aims at investigating photoluminescence from silicon nanocrystals within such a matrix.