Matrix spectral factorization is traditionally described as finding spectral factors having a fixed analytic pole configuration. The classification of spectral factors then involves studying the solutions of a certain algebraic Riccati equation which parametrizes their zero structure. The pole structure of the spectral factors can be also parametrized in terms of solutions of another Riccati equation. We study the relation between the solution sets of these two Riccati equations and describe the construction of general spectral factors which involve both zeroand pole-flipping on an arbitrary reference spectral factor.