While measures of affect are often included in studies using experience sampling methodology (ESM), the affective structure at the between- and within-person level has not been thoroughly investigated. Additionally, it is currently unclear to what extent this structure is influenced by variations in sampling protocols. Our first aim was to investigate the structure of affect at the between- and within-person level when measured with 8 or 18 affect items. Second, we investigated the invariance of this structure across different sampling frequencies and questionnaire lengths. Further, the reliability of positive and negative affect (PA and NA, respectively) was calculated. Participants (N = 163) were randomly assigned to receive either a 30 or 60 item questionnaire three, six, or nine times per day over 14 days. Momentary affect was assessed with 8 or 18 items. At both levels, a two-factor structure with correlated PA and NA showed the best fit compared to an orthogonal and a unidimensional model. However, the CFAs and additional EFAs indicated that items shared additional variance that could be accounted for by freeing residual correlations or adding more nuanced affect factors to achieve acceptable fit. A structure accounting for these additional relationships was invariant across different sampling protocols at the within-person level and resulted in high reliability. We observed differences between levels in the strength of the loadings, the correlation between PA and NA, and the number of factors suggested in EFAs. Taken together, these structural differences indicate a more discrete affective structure within than between persons