While recent breakthroughs in quantum computing promise the nascence of the quantum information age, quantum states remain delicate to control. Moreover, the required energy budget for large scale quantum applications has only sparely been considered. Addressing either of these issues necessitates a careful study of the most energetically efficient implementation of elementary quantum operations. In the present analysis, we show that this optimal control problem can be solved within the powerful framework of quantum speed limits. To this end, we derive state-independent lower bounds on the energetic cost, from which we find the universally optimal implementation of unitary quantum gates, for both single and N -qubit operations.