L12-Al3X (X = Li, Sc, and Zr) precipitates are the main strengthened phases of high-strength aluminum alloys and are critical for aerospace structural materials. Point defects and substitutional ternary elements change the mechanical properties of Al3X. In this paper, the effect of point defects, including vacancy, antisite, and substitutional element addition defects on the elastic modulus of the off-stoichiometric Al3X (X = Li, Sc, and Zr) phase were investigated by using first-principle calculations. The formation enthalpies of the defective Al3X alloy and isolated point defects in Al3X were calculated, and the results showed that the defects have an effect on the structure and elasticity of the off-stoichiometric Al3X phases. The lattice distortion, elastic constants, and elastic moduli were further investigated. It was found that the point defects increased the Young’s modulus for Al3Zr, and the doping of Er improved the Young’s modulus for off-stoichiometric Al3Li and Al3Sc. Adjusting the position of vacancies can improve the elastic modulus. In addition, the doping of substitutional elements (especially Sc, Ti, Zr, Hf, Ta, Mn, Ir, and Cf) can greatly increase the Young’s modulus of off-stoichiometric Al3Li.