Plasmonic structures possess rich physics related to the sensitivity of plasmon resonance to the change in the environmental dielectric constant, the enhanced light scattering and optical extinction, and the local field enhancement enabled strong light-matter interactions, which have been applied in refractive-index sensors, optical feedback in various micro-or nano-cavity lasers, surface enhanced Raman scattering spectroscopy, and high-sensitivity molecular detection. However, ultrafast optical response is another important aspect of plasmons, which can be utilized to achieve switching of optical signals in different spectral bands. These optical switching designs are very important for applications in optical logic circuits and optical communication system. In this review, we summarize a series of reports on ultrafast plasmonic optical switches, where we focus our discussions on the structural and device designs, instead of on their physics. By categorizing the designs of optical switches into different groups by their featured performances, we intend to propose the development trend and the commonly interested mechanisms of such ultrafast optical switches. We hope this review will supply helpful concepts and technical approaches for further development and new applications of ultrafast optical switching devices.