The energy parameter of the driving source for rock fragmentation by high-voltage pulsed discharge (RHPD) determines the effectiveness of rock fragmentation. The mapping relationship between the energy parameter of the RHPD driving source and the stress parameter is established, and an energy optimization design criterion for the RHPD driving source is proposed. The time-varying impedance of the plasma channel is simulated, and the spatial and temporal distribution of stress within the rock under the channel pressure loading is analyzed. A rock fragmentation criterion based on the stress impulse criterion is proposed, and a nonlinear constrained mathematical model for the energy optimization design for the RHPD driving source is developed. Using the existing driving source and load parameters as examples, the feasibility and accuracy of the proposed energy optimization design criterion for the RHPD driving source are verified, which provides theoretical guidance for the design of the RHPD driving source.