Vibration and noise pollution is one of the main pollution in modern society. In order to obtain good vibration damping effect, more effective phononic crystals should be designed. Based on the theory of elastic wave propagation in solids, three types of phononic crystals are designed in this paper using tungsten blocks embedded in a rubber matrix, which are cup-shaped phononic crystal, solid cylindrical phononic crystal, and hollow cylindrical phononic crystal. Firstly, the band gap characteristics and vibration losses of the three phononic crystals are analyzed by using the finite element method. Secondly, the physical mechanism of band gap formation is explored by vibration modes. Finally, the cup-shaped phononic crystal was introduced into the core layer of the sandwich plate to form the cup-shaped phononic crystal sandwich plate, and its vibration damping performance was analyzed. The results show that the three phononic crystals can form three band gaps in the range of 0–800 Hz, and the first low-frequency band gap starts at about 140 Hz and is all wider than 200 Hz. It is noteworthy that the average loss of vibration transmission of the three types of phononic crystals is more than 69 dB, which possesses stronger damping capability and wider low- and middle-frequency band gaps than the flat plate type phononic crystals. The vibration direction of the phononic crystal is at an angle of 90° to the wave vector, which prevents the propagation of elastic waves. The phononic crystal embedded in the sandwich panel plate is more in line with the actual demand of vibration damping and obtains good vibration damping effect. The research in this paper can provide more feasibility for phononic crystal damping.