The increasing penetration of distributed generation (DG) to power distribution networks mainly induces weaknesses in the sensitivity and selectivity of protection systems. In this manner, conventional protection systems often fail to protect active distribution networks (ADN) in the case of short-circuit faults. To overcome these challenges, the accurate detection of faults in a reasonable fraction of time appears as a critical issue in distribution networks. Machine learning techniques are capable of generating efficient analytical expressions that can be strong candidates in terms of reliable and robust fault detection for several operating scenarios of ADNs. This paper proposes a deep group method of data handling (GMDH) neural network based on a non-pilot protection method for the protection of an ADN. The developed method is independent of the DG capacity and achieves accurate fault detection under load variations, disturbances, and different high-impedance faults (HIFs). To verify the improvements, a test system based on a real distribution network that includes three generators with a capacity of 6 MW is utilized. The extensive simulations of the power network are performed using DIgSILENT Power Factory and MATLAB software. The obtained results reveal that a mean absolute percentage error (MAPE) of 3.51% for the GMDH-network-based protection system is accomplished thanks to formulation via optimized algorithms, without requiring the utilization of any feature selection techniques. The proposed method has a high-speed operation of around 20 ms for the detection of faults, while the conventional OC relay performance is in the blinding mode in the worst situations for faults with HIFs.