Conformational space annealing (CSA), a global optimization method, has been applied to various protein structure modeling tasks. In this paper, we applied CSA to the cryo‐EM structure modeling task by combining the python subroutine of CSA (PyCSA) and the fast relax (FastRelax) protocol of PyRosetta. Refinement of initial structures generated from two methods, rigid fitting of predicted structures to the Cryo‐EM map and de novo protein modeling by tracing the Cryo‐EM map, was performed by CSA. In the refinement of the rigid‐fitted structures, the final models showed that CSA can generate reliable atomic structures of proteins, even when large movements of protein domains were required. In the de novo modeling case, although the overall structural qualities of the final models were rather dependent on the initial models, the final models generated by CSA showed improved MolProbity scores and cross‐correlation coefficients to the maps. These results suggest that CSA can accomplish flexible fitting and refinement together by sampling diverse conformations effectively and thus can be utilized for cryo‐EM structure modeling.