This study employs first-principles calculations to investigate how introducing Yb into aluminum nitride (AlN) leads to a large enhancement in the material’s piezoelectric response (d33). The maximum d33 is calculated to be over 100 pC/N, which is 20 times higher than that of AlN. One reason for such a significant improvement in d33 is the elastic-softening effect, which is indicated by a decrease in the elastic constant, C33. The strain sensitivity (du/dε) of the internal parameter, u, is also an important factor for improving the piezoelectric stress constant, e33. On the basis of mixing enthalpy calculations, YbxAl1−xN is predicted to be more stable as a wurtzite phase than as a rock salt phase at composition up to x ≈ 0.7. These results suggest that Yb can be doped into AlN at high concentrations. It was also observed that the dielectric constant, ε33, generally increases with increasing Yb concentrations. However, the electromechanical coupling coefficient, k332, only increases up to x = 0.778, which is likely because of the relatively lower values of ε33 within this range.