Recent experimental work on superconducting transmon qubits in three-dimensional (3D) cavities shows that their coherence times are increased by an order of magnitude compared to their two-dimensional cavity counterparts. However, to take advantage of these coherence times while scaling up the number of qubits it is advantageous to address individual qubits which are all coupled to the same 3D cavity fields. The challenge in controlling this system comes from spectral crowding, where the leakage transition of qubits is close to computational transitions in other qubits. Here, it is shown that fast pulses are possible which address single qubits using two-quadrature control of the pulse envelope, while the derivative removal by adiabatic gate method of Motzoi et al. [Phys. Rev. Lett. 103, 110501 (2009)] alone only gives marginal improvements over the conventional Gaussian pulse shape. On the other hand, a first-order result using the Magnus expansion gives a fast analytical pulse shape which gives a high-fidelity gate for a specific gate time, up to a phase factor on the second qubit. Further numerical analysis corroborates these results and yields to even faster gates, showing that leakage-state anharmonicity does not provide a fundamental quantum speed limit.