The anti-cracking properties of polymer-modified asphalt depend largely on the molecular structure of the polymer modifier. However, the mysterious structure-performance relationship is still elusive. In this paper, three kinds of polymers with different chain structures were selected to address this issue. The indices of styrene, trans-butadiene, aliphatic branched-chain, and aliphatic long-chain from the infrared spectrum were used to quantify the functional group compositions of polymer modifiers. Viscoelastic parameters, including relaxation time, dissipation energy ratios, and stiffness were assessed to illustrate the anti-cracking properties of polymer-modified asphalt. Results showed that relaxation time and dissipation energy ratios were mainly determined by the polymer network strength, molecular size, aliphatic chain feature, and the orientations speed of aliphatic chains. The short relaxation time and high dissipation ratio lead to the low stiffness and favorable low-temperature performance of asphalt. The improvement of these performances requires a polymer with high indices of an aliphatic long-chain, styrene, aliphatic branched-chain, and trans-butadiene, respectively. An aliphatic-long chain, aliphatic branched-chain, and trans-butadiene were soft segments in asphalt while styrene was the rigid segment. The soft segments affect the intramolecular friction, orientation, and thermal motion at low temperatures, whereas the rigid segment enhances the strength of polymer networks. Thus, the anti-cracking property of polymer-modified asphalt can be improved by adjusting the ratio of soft and rigid segments in the polymer modifier.