The human intruder test is a testing paradigm designed to measure rhesus macaques’ behavioral responses to a stressful and threatening situation. In the test, an unfamiliar human positions him/herself in various threatening positions relative to a caged macaque. This paradigm has been utilized for over twenty years to measure a variety of behavioral constructs, including fear and anxiety, behavioral inhibition, emotionality, and aggression. To date there have been no attempts to evaluate comprehensively the structure of the behavioral responses to the test. Our first goal was to identify the underlying latent factors affecting the different responses among subjects, and our second goal was determine if rhesus reared in different environments respond differently in this testing paradigm. To accomplish this, we first performed exploratory and confirmatory factor analyses on the behavioral responses of 3–4 month-old rhesus macaques, utilizing data from over 2,000 separate tests conducted between 2001–2007. Using the resulting model, we then tested to see whether early rearing experience affected responses in the test. Our first analyses suggested that most of the variation in infant behavioral responses to the human intruder test could be explained by four latent factors: “Activity,” “Emotionality,” “Aggression,” and “Displacement.” Our second analyses revealed a significant effect of rearing condition for each factor score (P < 0.001); most notable socially-reared animals had the lowest Activity score (P < 0.001), indoor mother-reared animals had the highest Displacement score (P < 0.001), and nursery-reared animals had the highest Emotionality (P < 0.001) and lowest Aggression scores (P < 0.001). These results demonstrate that this standardized testing paradigm reveals multiple patterns of response, which are influenced by an animal’s rearing history.