How is cognitive empathy related to sociality, imagination, and other psychological constructs? How is it altered in disorders of human social cognition? We leveraged a large data set (1,168 students, 62% female) on the Reading the Mind in the Eyes test (RMET), the Autism Quotient (AQ), and the Schizotypal Personality Questionnaire (SPQ-BR) to test the hypotheses that the RMET, as a metric of cognitive empathy, reflects mainly social abilities, imagination, or both. RMET showed the expected female bias in performance, though only for eyes that expressed emotions and not for neutral expressions. RMET performance was significantly, and more strongly, associated with the AQ and SPQ subscales that reflect aspects of imagination (AQ-Imagination and SPQ-Magical Ideation) than aspects of social abilities (AQ-Social, AQ-Communication, and SPQ-Interpersonal subscales). These results were confirmed with multiple regression analysis, which also implicated increased attention (AQ-Attention Switching and, marginally non-significantly, AQ-Attention to Detail) in RMET performance. The two imagination-related correlates of RMET performance also show the strongest sex biases for the AQ and SPQ: male biased in AQ-Imagination, and female biased in SPQ-Magical Ideation, with small to medium effect sizes. Taken together, these findings suggest that cognitive empathy, as quantified by the RMET, centrally involves imagination, which is underdeveloped (with a male bias) on the autism spectrum and overdeveloped (with a female bias) on the schizotypy spectrum, with optimal emotion-recognition performance intermediate between the two. The results, in conjunction with previous studies, implicate a combination of optimal imagination and focused attention in enhanced RMET performance.