To provide insight into the mechanical behavior and microstructural evolution of bulk nanograined (NG) Ni-based alloys during annealing, the Ni-based alloy sheets with grain size about 50 nm was produced through severe cold-rolling at room temperature, and then the cold rolled (CRed) Ni-based alloys were annealed at different states. The evolution of the nanostructure of the CRed Ni-based alloy during annealing and corresponding change in mechanical properties was investigated. The results showed that the CRed Ni-based alloy exhibited prominent enhancement in the yield strength (YS), ultimate tensile strength (UTS), which increased respectively from 253 MPa to 1455 MPa, 684 MPa to 1557 MPa. Further increase of the YS and UTS were obtained in the annealed-CRed Ni-based alloy with dual-phase. The YS and UTS of the NG dual-phase Ni-based alloy was respectively 2013 MPa and 2061MPa, which was annealed at 700 °C for 1h. In terms of the microstructural evolution, lower density of defects on the grain boundary were observed and the nanograins can be maintained about 100 nm even when annealed for 30 h at 700 °C, which suggests high thermal stability at this temperature. Both the high thermal stability and strength are due to the formation of the γ′ precipitates and slight grain growth of the NG matrix.