Are Taiwanese voters polarized? By presenting four novel visualizations based on a factor analysis of Taiwan National Security Surveys from 2002 to 2017, this article describes the contours of structural change in Taiwan politics. Overall, the cross-strait position among Taiwanese voters can be described by a stable inverted U shape over time. This arises from the fact that most nonpartisans—typically neglected in the literature on polarization—are moderate. Before 2008, increasing polarization among partisans can be attributed to pan-green voters moving toward independence. Between 2008 and 2014, decreasing polarization stems from moderates self-identifying as pan-blue supporters. Since 2014, a record-breaking number of nonpartisans have left the pan-blue camp, and more extreme pan-blue voters have contributed to a return of polarization among partisans. The results yield important implications for the study of polarization and populism, as well as for the future of Taiwanese politics.