When people interact with novel technologies (e.g., robots, novel technological tools), the word “creepy” regularly pops up. We define creepy situations as eliciting uneasy feelings and involving ambiguity (e.g., on how the behave or how to judge the situation). A common metric for creepiness would help evaluating creepiness of situations and developing adequate interventions against creepiness. Following psychometrical guidelines, we developed the Creepiness of Situation Scale (CRoSS) across four studies with a total of N = 882 American and German participants. In Studies 1–3, participants watched a video of a creepy situation involving technology. Study 1 used exploratory factor analysis in an American sample and showed that creepiness consists of emotional creepiness and creepy ambiguity. In a German sample, Study 2 confirmed these subdimensions. Study 3 supported validity of the CRoSS as creepiness correlated positively with privacy concerns and computer anxiety, but negatively with controllability and transparency. Study 4 used the scale in a 2 (male vs. female experimenter) × 2 (male vs. female participant) × 2 (day vs. night) field study to demonstrate its usefulness for non-technological settings and its sensitivity to theory-based predictions. Results indicate that participants contacted by an experimenter at night-time reported higher feelings of creepiness. Overall, these studies suggest that the CRoSS is a psychometrically sound measure for research and practice.