The focus of this article is to present an effective anomaly detection model for an encrypted network session by developing a novel IP reputation scoring model that labels the incoming session IP address based on the most similar IP addresses in terms of both network and geo-contextual knowledge. We provide empirical evidence that considering not only traditional network information but also geo-contextual information provides better threat assessment. The reputation scores provide a means to quantitatively capture good and bad IP behavior, making our model ideal for detecting malicious network behavior. With network encryption being the most practical solution to data security and privacy today, our approach expands the network administrator's ability to make decisions about IP addresses' trustworthiness in an encrypted session with limited network information.