The high-temperature properties of new alloys need to be investigated to guide the hot working process. The temperature sensitivity of various microstructures of Fe45Mn15Cr15Ni25 and Fe35Mn15Cr15Ni25Al10 cobalt-free high-entropy alloys was investigated using high-temperature tensile tests. For recrystallized alloys, the increase in aluminum (Al) atoms exacerbates the emergence of serration behavior, prolongs the strain hardening capacity, and delays the decrease in plasticity. The Fe35Mn15Cr15Ni25Al10 alloy, with a high-density precipitated phase, exhibits excellent mechanical properties at 673 K. It has a yield strength of 735 MPa, an ultimate tensile strength of 1030 MPa, and an elongation of 11%. Ultimately, it has been found that the addition of the element Al improves the strength, oxidation resistance, and thermal stability of the alloy. According to the solid solution strengthening model fitting and nanoindentation results, the temperature sensitivity of the yield strength of the alloy is primarily attributed to the solid solution strengthening and phase interface forces. There is relatively less variation in grain boundary strengthening and precipitation strengthening. The relationship between the mechanical properties and temperature of the alloy can be predicted to guide the machining process of the alloy.