Train derailments in railway switches are becoming more and more common, which have caused serious casualties and economic losses. Most previous studies ignored the derailment mechanism when vehicles pass through the turnout. With this consideration, this work aims to research the 3D derailment coefficient limit and passing performance in turnouts through the quasi-static analysis and multi-body dynamic simulation. The proposed derailment criteria have considered the influence of creep force and wheelset yaw angle. Results show that there are two derailing stages in switch panel, which are climbing the switch rail and stock rail, respectively. The 3D derailment coefficient limit at the region of top width 5 mm to 20 mm is much lower than the main track rail, which shows that wheels are more likely to derail in this area. The curve radius before the switch rail is suggested to be set as 350 m. When the curve radius before turnout is 65 m, the length of the straight line between the curve and turnout needs to be larger than 3 m. This work can provide a good understanding of the derailment limit and give guidance to set safety criteria when vehicles pass through the turnout.