We theoretically investigate the stability of a rectangular skyrmion crystal without fourfold rotational symmetry under an orthorhombic distortion in centrosymmetric tetragonal magnets. The results are obtained by numerically simulated annealing for an effective spin model with competing interactions in momentum space and magnetic anisotropy. By constructing the low-temperature phase diagram while changing the interaction ratio arising from the orthorhombic distortion, we find that the rectangular skyrmion crystal remains stable in an external magnetic field against distortion. We show that the degree of fourfold rotational symmetry tends to recover when the magnetic field is increased. The relevance to the skyrmion-hosting material EuAl4 is also discussed.