The analysis of factor structures is one of the most critical psychometric applications. Frequently, variables (i.e., items or indicators) resulting from questionnaires using ordinal items with 2 to 7 categories are used. There are plenty of articles that recommend treating ordinal variables in a factor analysis by default as ordinal and not as continuous imposing a multivariate normal distribution assumption. In this article, we critically reflect that the reasoning behind such suggestions is flawed. In our view, findings from simulation studies cannot tell about the right modeling strategy of ordinal variables in factor analysis. Moreover, it is argued that ordinal factor models impose a normality assumption for underlying continuous variables, which might also often be false in empirical applications. However, researchers seldom opt for these much-needed, more flexible modeling strategies if the ordinal nature variables would seriously be modeled in factor analysis. Finally, the consequences of modeling choices for validity, reliability, measurement invariance, and the assessment of global model fit are discussed.