Various leak detection methods have been developed for water distribution networks (WDNs). Since sufficient tools are already available, creating further models draws less attention over time. Instead, existing methods should be evaluated across various WDNs to examine the impact of network characteristics on the detectable leak size (QDL). This study tests a convolutional neural network-based leak detection model for 12 WDNs. Each network has different characteristics such as system demand, pipe diameter and length, and topology. Seven leak sizes are evaluated per network, and two detection performance metrics, namely, the detection probability (DP) and rate of false alarms (RF), are calculated to assess QDL. Additionally, a new metric is derived to simultaneously evaluate DP and RF. The results indicate that QDL varies substantially depending on the system type, either transmission-oriented or distribution-oriented. Identifying leaks for the former system is challenging, whereas the latter exhibits high DPs with acceptable number of false alarms when detecting small leaks (e.g., 1 L/s). Moreover, QDL is more sensitive to hydraulics than topological characteristics (e.g., branch index). The combination of network parameters in the energy loss equation provided the most suitable relationship with leak detection.