Fracture of viscoplastic materials is a complex process due to its time-dependent and plastic responses. Numerical simulation for fracture plays a significant role in crack prediction and failure analysis. In recent years, the phase field model is becoming a competitive approach to predict crack growth and has been extended to inelastic materials such as elasto-plastic, viscoelastic and viscoplastic materials, etc. However, the contribution of inelastic energy to crack growth is seldom studied. For this reason, we implement the phase field model coupled with a viscoplastic constitutive in finite element framework, in which the elastic energy and inelastic energy are used as crack driving energy. The implicit algorithm for a viscoplastic constitutive is presented, this procedure is suitable for other viscoplastic constitutive relations. The strain rate effect, creep effect, stress relaxation effect and cyclic loading responses are tested using a single element model with different inelastic energy contributions; A titanium alloy plate specimen and a stainless-steel plate specimen under tension are studied compared with the experimental observations in existing literature. The results show that the above typical damage phenomenon and fracture process can be well reproduced; The inelastic energy significantly affects the force-displacement responses but has slight effect on the crack paths.