The thermal expansion and the phase transition of Fe-15.6 wt. %Co-12 wt. %Ni single-phase solid solution alloy were systematically investigated by thermal analysis experiments and molecular dynamics simulations. The coefficient of thermal expansion (CTE) was accurately measured in the temperature range of 300-1580 K. The eccentric changes of thermal expansion ranging from 900 to 1150 K were verified from the incomplete transformation of α-Fe phase to γ-Fe phase by differential scanning calorimetry (DSC) and in situ X-ray diffraction experiments. The CTE of α-Fe phase increases nonlinearly from 9.29 × 10−6 to 1.278 × 10−5 K−1 in the range of 300-900 K, which is in good agreement with the results obtained by molecular dynamics simulation, whereas the CTE of γ-Fe phase increases linearly from 2.024 × 10−5 to 2.398 × 10−5 K−1 in the range of 1150-1580 K. Meanwhile, the visual atomic positions at different temperatures indicate that thermal expansion is attributed to the anharmonic vibration and short-range diffusion of atoms when the temperature exceeds a certain value. Furthermore, the Curie temperature is determined as 725 K by the thermal expansion and DSC experiments. Additionally, the isothermal sections of the Fe-rich corner [Fe-5x wt. %Co-5y wt. %Ni(2 ≤ x + y ≤ 8)] in Fe-Co-Ni non-equilibrium ternary phase diagram at 300 K are derived by X-ray diffraction. Moreover, the CTE ranging from 300 to 1700 K of the Fe-rich corner in Fe-Co-Ni ternary phase diagram was predicted theoretically on the basis of the molecular dynamics method.