Recent work has shown that the idiosyncrasies of the observer can contribute more to the variance of social judgments of faces than the features of the faces. However, it is unclear what conditions determine the relative contributions of idiosyncratic and shared variance. Here, we examine three conditions: type of judgment, response scales, and diversity of face stimuli. First, we show that for simpler, directly observable judgments (e.g., masculinity) shared exceeds idiosyncratic variance, whereas for more complex, less directly observable judgments (e.g., trustworthiness) idiosyncratic exceeds shared variance. Second, dichotomous forced-choice responses (i.e., “yes”/”no”) resulted in greater shared variance compared to multi-point Likert- type responses. Third, we show that judgments of more diverse face images increase the amount of shared variance. Finally, using machine learning methods, we examine how stimulus (e.g., emotion resemblance, skin luminosity) and observer variables (e.g., race, age) contribute to shared and idiosyncratic variance of judgments. Overall, our results indicate that an observer's age is the most consistent and best predictor of idiosyncratic variance contributions to face judgments measured in the current research.