The aim of this paper is to show that there can be either only one or uncountably many contexts in any spectral effect algebra, answering a question posed in [S. Gudder, Convex and Sequential Effect Algebras, (2018), arXiv:1802.01265]. We also provide some results on the structure of spectral effect algebras and their state spaces and investigate the direct products and direct convex sums of spectral effect algebras. In the case of spectral effect algebras with sharply determining state space, stronger properties can be proved: the spectral decompositions are essentially unique, the algebra is sharply dominating and the set of its sharp elements is an orthomodular lattice. The article also contains a list of open questions that might provide interesting future research directions.