The main sub-harmonic joint resonance of the van der Pol-Duffing system with a quintic oscillator under dual-frequency excitation is investigated in this paper. The study examines the conditions for chaos and vibration resonance under different parameters. An approximate analytical solution for the principal sub-harmonic joint resonance of the system under dual-frequency excitation is obtained using the multi-scale method, while the Melnikov method provides necessary conditions for chaos in the system. Furthermore, based on the fast and slow variable separation method, vibration resonance of the system under various conditions is determined. Numerical simulations explore amplitude-frequency characteristics of total response at different excitation frequencies through analytical and simulation methods, with consistency between numerical and analytical results verified by plotting amplitude-frequency characteristic curves. Additionally, an analysis is conducted to investigate how fractional order, fractional differential coefficient, and cubic stiffness affect co-amplitude-frequency curves of the van der Pol-Duffing oscillator. The analysis reveals that a jump phenomenon exists in co-amplitude-harmonic resonance of this oscillator; moreover, changes in different parameters can alter both jump points and cause disappearance of such phenomena. Sub-critical fork bifurcation behavior as well as supercritical fork bifurcation behavior are studied along with vibration resonance caused by parameter variations. Results indicate that sub-critical fork bifurcation arises from changes in excitation term coefficient while supercritical fork bifurcation occurs due to fractional order variations. Furthermore, when different fractional order values are considered, there will be changes in resonance location, response amplitude gain, and vibration resonance mode within the system. The implementation of this measure enhances our comprehension of the vibration characteristics of the system, thereby refining the accuracy of the model and bolstering the stability of the system. Additionally, it serves as a preventive measure against resonance issues, which are particularly critical for mitigating the hazards associated with system resonance triggered by supercritical fork bifurcations. These hazards encompass potential structural damage and equipment failure.