Viral infection severity often varies with host factors such as age and sex. The pathogenesis of infections caused by a broad range of viruses, from neurotropic viruses like Rabies and Zika to respiratory viruses such as influenza and SARS-CoV-2, differ between the sexes and across the lifespan. Typically, older males are more susceptible to severe acute outcomes, while females are more vulnerable to the post-acute sequelae of infections. All of these complications can include neuroinflammation, stroke, cognitive dysfunction, and delirium. While these symptoms can be secondary to infection, recent studies suggest that even peripheral infections can lead to neuropathological changes in the brain. However, few studies have characterized the expression of viral receptors in the human brain or examined age- or sex-related differences in such expression. In this study, we used a publicly accessible transcriptomic database to assess the impact of age and sex on the expression of 67 viral host factor genes, associated with ten virus families. Analyzing data from 15 brain areas (n=33, F=14, M=19, age:4 mo-80 yrs), we determined the lifespan trajectory for each gene in each area via LOESS regressions. We used unsupervised hierarchical clustering to determine if a brain-wide pattern or virus family pattern can be detected. Using Dense-tSNE, a dimension-reduction and visualization technique, we discovered four distinct developmental trajectories, clustering the areas into two mixed-sex subcortical clusters and one each of male and female cortical clusters. Applying Differential Expression Sliding Window Analysis (DeSWAN), we identified the genes driving these age- and sex-related differences. Many sex differences were noted in childhood, potentially impacting the brain’s susceptibility to viral infections and underscoring a broader dimorphic organization of male and female brains. These insights contribute to our understanding of sex-specific responses to viral infections, offering the potential for more personalized treatment strategies.