We present a general theoretical framework for finding the time-optimal unitary evolution of the quantum systems when the Hamiltonian is subject to arbitrary constraints. Quantum brachistochrone (QB) is such a framework based on the variational principle, whose drawback is that it only deals with equality constraints. While inequality constraints can be reduced to equality ones in some situations, they usually cannot, especially when a drift field, an uncontrollable part, is present in the Hamiltonian. We first develop a framework based on Pontryagin’s maximum principle (MP) in order to deal with inequality constraints as well. The new framework contains QB as a special case, and their detailed correspondence is given. Second, we address the problem of singular controls, which satisfy MP trivially so as to cause a trouble in determining the optimal protocol. To overcome this difficulty, we derive an additional necessary condition for a singular protocol to be optimal by applying the generalized Legendre–Clebsch condition. Third, we discuss general relations among the drift, the singular controls, and the inequality constraints. Finally, we demonstrate how our framework and results work in some examples. We also discuss the physical meaning of singular controls.