As the most fascinating, complex, and dynamic part of our organism, the human brain is shaped by many interacting factors that not only are of neurobiological (including sex hormones) and environmental origin but are also sociocultural in their very nature (such as social roles). Gender is one of these factors. Most neurological, neurodevelopmental, neuropsychiatric, and psychosomatic disorders are characterized by impairments in visual social cognition (primarily body language reading and face perception) and a skewed sex ratio: females and males are affected differently in terms of clinical picture, prevalence, and severity. Is the social brain sex specific? This is still an open question. For a long time and for many reasons, sex differences have been overlooked or entirely ignored in neuroscience and biomedical research: there is a paucity of neuroimaging work examining sex differences in the social brain. However, the pattern of experimental behavioral data in both healthy, typically developing individuals and patients with deficient social cognition is beyond simple interpretation: contrary to popular wisdom, females are not always more proficient in understanding social signals, and their social abilities may be particularly affected by disease. Clarification of how neurobiological sex and sociocultural gender affect the social brain would provide novel insights into understanding gender‐specific vulnerability to neuropsychiatric disorders. This interaction is far beyond simplicity. Although sex differences represent a rather delicate topic, underestimation or exaggeration of possible effects retards progress in the field. © 2016 Wiley Periodicals, Inc.