Sexuality is an integral part of human nature, yet we know little about its origins and underlying mechanisms. Understanding how the brain processes sexual stimuli is crucial for advancing our knowledge of the origins of sexuality and its variations. Only if a stimulus was internally evaluated as ‘attractive’ the information can progress to evoke sexual desire or arousal. Importantly, such evaluation processes require an internal reference against which external information is compared. These processes, however, remain largely unexplored, and hence we still do not understand why we are attracted to what we are attracted to. By synthesizing the existing literature and integrating existing models, this narrative review proposes a novel neuroscientific framework on sexuality, addressing key theoretical gaps. As a result, the concept of sexual priors in the brain is introduced. Within this framework, sexual priors refer to mental ‘images’ representing a collection of internally stored information of what we regard as sexually attractive. Such mental representations act as reference in early internal evaluation processes of sexual stimuli. It is suggested that sexual priors are (partly) innate and sex-specific, contributing to heterosexuality. Evolutionary aspects that could explain the development of these priors are discussed. Furthermore, the framework provides possible explanations for variations in sexual orientation through potential cross-sex shifts in sexual prior content. The refinement of sexual prior content over time may explain the diversity of sexual interests and attraction patterns among humans. Moreover, the concept can be applied to a variety of psychiatric conditions related to atypical and maladaptive sexual behaviors. A proposition is put forward regarding brain areas and networks that may be involved in the formation of sexual priors, serving as potential targets for future research. In summary, by integrating sexual priors into models of sexual stimuli processing, this article fills theoretical gaps while inspiring further research. This work aims to enhance understanding of sexuality, which ultimately could increase social awareness, foster tolerance, and promote psychological well-being.