Indirect control of qubits by a quantum actuator has been proposed as an appealing strategy to manipulate qubits that couple only weakly to external fields. While universal quantum control can be easily achieved when the actuator-qubit coupling is anisotropic, the efficiency of this approach is less clear. Here we analyze the time efficiency of quantum actuator control. We describe a strategy to find time-optimal control sequences by the quantum actuator and compare their gate times with direct driving, identifying regimes where the actuator control performs faster. As a paradigmatic example, we focus on a specific implementation based on the nitrogen-vacancy center electronic spin in diamond (the actuator) and nearby 13 C nuclear spins (the qubits).