Given a projective irreducible symplectic manifold M of dimension 2n, a projective manifold X and a surjective holomorphic map f : M → X with connected fibers of positive dimension, we prove that X is biholomorphic to the projective space of dimension n. The proof is obtained by exploiting two geometric structures at general points of X: the affine structure arising from the action variables of the Lagrangian fibration f and the structure defined by the variety of minimal rational tangents on the Fano manifold X.