We investigate the lattice CP N −1 sigma model on S 1 s (large) × S 1 τ (small) with the Z N symmetric twisted boundary condition, where a sufficiently large ratio of the circumferences (L s L τ) is taken to approximate R × S 1. We find that the expectation value of the Polyakov loop, which is an order parameter of the Z N symmetry, remains consistent with zero (| P | ∼ 0) from small to relatively large inverse coupling β (from large to small L τ). As β increases, the distribution of the Polyakov loop on the complex plane, which concentrates around the origin for small β, isotropically spreads and forms a regular N-sided-polygon shape (e.g. pentagon for N = 5), leading to | P | ∼ 0. By investigating the dependence of the Polyakov loop on S 1 s direction, we also verify the existence of fractional instantons and bions, which cause tunneling transition between the classical N vacua and stabilize the Z N symmetry. Even for quite high β, we find that a regular-polygon shape of the Polyakov-loop distribution, even if it is broken, tends to be restored and | P | gets smaller as the number of samples increases. To discuss the adiabatic continuity of the vacuum structure from another viewpoint, we calculate the β dependence of "pseudo-entropy" density ∝ T xx − T τ τ. The result is consistent with the absence of a phase transition between large and small β regions.