Medium manganese (medium-Mn) steel, one of the third-generation advanced high-strength steels (AHSS), delivers impressive mechanical properties such as high yield strength, ultimate tensile strength, and uniform elongation. One notable feature of medium-Mn steels is the presence of ultrafine-grained (UFG) austenite, achieved through phase transformation from the parent martensite phase during intercritical annealing. While, in general, UFG is considered a strengthening mechanism, the impact of UFG austenites in medium-Mn steel has not been fully studied. In this manuscript, we advance our previous work on crystal plasticity simulation based on the Taylor model to consider fully resolved high-fidelity microstructures and systematically study the influence of the UFG austenites. The original microstructure with UFG is reconstructed from a set of serial electron backscatter diffraction (EBSD) scans, where the exact grain morphology, orientation, and phase composition are preserved. This microstructure was further analyzed to identify the UFG austenites and recover them to their parent martensite before the intercritical annealing. These two high-fidelity microstructures are used for a comparative study using dislocation density-based crystal plasticity finite modeling to understand the impact of UFG austenites on both the local and overall mechanical responses.