Delay Tolerant Network (DTN) is developed to cope with intermittent connectivity and long delay in wireless networks. Due to the limited connectivity, DTN is vulnerable to blackhole and greyhole attacks in which malicious nodes intentionally drop all or part of the received messages. Although existing proposals could accurately detect the attack launched by individuals, they fail to tackle the case that malicious nodes cooperate with each other to cheat the defense system. In this paper, we suggest a scheme called Statistical-based Detection of Blackhole and Greyhole attackers (SDBG) to address both individual and collusion attacks. Nodes are required to exchange their encounter record histories, based on which other nodes can evaluate their forwarding behaviors. To detect the individual misbehavior, we define forwarding ratio metrics that can distinguish the behavious of attackers from normal nodes. Malicious nodes might avoid being detected by colluding to manipulate their forwarding ratio metrics. To continuously drop messages and promote the metrics at the same time, attackers need to create fake encounter records frequently and with high forged numbers of sent messages. We exploit the abnormal pattern of appearance frequency and number of sent messages in fake encounters to design a robust algorithm to detect colluding attackers. Extensive simulation shows that our solution can work with various dropping probabilities and different number of attackers per collusion at high accuracy and low false positive.