The seepage characteristics of clay core walls are crucial for the seepage safety of core rockfill dams, with the permeability coefficient in the unsaturated zone being nonlinear. To accurately determine the unsaturated seepage parameters in clay core rockfill dams, this paper first proposes an enhanced reptile search algorithm (ERSA) by applying three improvement strategies: Arnold’s cat chaotic map, nonlinear evolutionary factor, and adaptive Cauchy–Gaussian mutation with variable weight. Then, by integrating the ERSA with the unsaturated seepage finite element method, an inverse modeling approach is developed. This approach is applied to an actual rockfill dam with operational monitoring data to determine the unsaturated seepage parameters of the clay core. Results indicate that the ERSA outperforms the original RSA in test functions, and the calculation results of the seepage parameters determined through inversion are consistent with the monitoring data, showing an overall mean absolute error of 1.086 m. The inverse modeling approach provides a valuable reference for determining unsaturated seepage parameters in similar clay core rockfill dams.