In this work, the dynamics of Duffing-Holmes oscillator with fractional order nonlinearity is explored. Basically, a fractional spatial derivative is introduced to the cubic term, and the order of the derivative α is varied between zero and two. The evolution of the dynamics of the system from nonlinear behavior to linear behavior is investigated using multiple tools such as phase portraits, Poincare maps, and bifurcation diagrams. We have demonstrated that as α increases the system can alternate between chaotic and periodic states depending on the parameters setting. However, the overall impact transforms the system into simpler dynamics and eventually causes the chaotic regions to fade out regardless of the system settings. The largest α at which the system still exhibits chaotic behavior is estimated to be around 1.17 and for transient chaos is estimated to be 1.25.