TaSe$_2$ has several different polytypes and abundant physical properties such as superconductivity and charge density waves (CDW), which had been investigated in the past few decades. However, there is no report on the physical properties of 4$Ha$ polytype up to now. Here we report the crystal growth and discovery of superconductivity in tantalum self-intercalated 4$Ha$-Ta$_{1.03}$Se$_2$ single crystal with a superconducting transition onset temperature $T_{\rm c}$ $\approx$ 2.7 K, which is the first observation of superconductivity in 4$Ha$ polytype of TaSe$_2$. A slightly suppressed CDW transition is found around 106 K. A large $\mu_0H_{\rm c2}/T_{\rm c}$ value of about 4.48 is found when magnetic field is applied in the $ab$ plane, which probably results from the enhanced spin-orbit coupling(SOC). Special stacking faults are observed, which further enhance the anisotropy. Although the density of states at the Fermi level is lower than that of other polytypes, $T_{\rm c}$ remains the same, indicating the stack mode of 4$Ha$ polytype may be beneficial to superconductivity in TaSe$_2$.