In our previous article [J. High Energ. Phys. 2023, 15 (2023)], we showed that the strictly massless spin-3/2 field, as well as the strictly and partially massless spin-5/2 fields, on N-dimensional (N ≧ 3) de Sitter spacetime (dSN) are non-unitary unless N=4. The (non-)unitarity was demonstrated by simply observing that there is a (mis-)match between the representation-theoretic labels that correspond to the Unitary Irreducible Representations (UIR's) of the de Sitter (dS) algebra spin(N,1) and the ones corresponding to the space of eigenmodes of the field theories. In this paper, we provide a technical representation-theoretic explanation for this fact by studying the (non-)existence of positive-definite, dS invariant scalar products for the spin-3/2 and spin-5/2 strictly/ partially massless eigenmodes on dSN (N ≧ 3). Our basic tool is the examination of the action of spin(N,1) generators on the space of eigenmodes, leading to the following findings. For odd N, any dS invariant scalar product is identically zero. For even N > 4, any dS invariant scalar product must be indefinite. This gives rise to positive-norm and negative-norm eigenmodes that mix with each other under spin(N,1) boosts. In the N=4 case, the positive-norm sector decouples from the negative-norm sector and each sector separately forms a UIR of spin(4,1). Our analysis makes extensive use of the analytic continuation of tensor-spinor spherical harmonics on the N-sphere (SN) to dSN and also introduces representation-theoretic techniques that are absent from the mathematical physics literature on half-odd-integer-spin fields on dSN.