Outlier detection in process mining refers to aspects such as infrequent behavior in relation to the underlying business process models or to anomalous latencies of task execution, termed as temporal anomalies. In this work, we focus on the latter form of anomalies and we aim at investigating in depth the behavior of several proximity-based variants, which are shown to outperform simple statistical ones. We investigate multiple distance functions and approaches to establishing the outlierness of traces or individual tasks, and we explain the superiority of our proposals over existing proximity and probability distribution fitting-based techniques yielding up to 2.05X higher F1 score. We also provide guidelines as to which variant to be chosen based on the type of anomalies targeted and the dataset characteristics.