In the milling process of composite materials, the initial chatter frequency is not obvious and is easily swamped by the rest of the signals, making frequency monitoring difficult, so the study proposes a chatter frequency monitoring method based on frequency cancellation algorithms and wavelet packet decomposition. The results of the experiments shown that the frequency cancellation algorithm can successfully remove invalid signals, such as spindle rotation frequency and cutter tooth frequency, and only keep the necessary signals, at which point the chattering frequency may be observed at a frequency of roughly 1333 Hz. The influence of the frequency bands s5, s9, s10, s12, and s13 after de-frequency removal should be avoided because they all have a low energy share of roughly 23 %, 0.9 %, 5 %, 10 %, and 16 %, respectively, and are less sensitive to chatter. For milling edge depths of 0.5 mm, 2 mm, and 4 mm, the average chatter thresholds were around 3.27, 2.9, and 2.89, respectively. It was challenging to pinpoint the chatter of the system because the empirical modal decomposition observed an average chatter energy entropy of just 1.55 and found that its fluctuations at the milling edge depth junction were insignificant. On the other hand, the chattering could be plainly seen since the energy entropy experienced a substantial and dramatic fluctuation at the intersection of the milling edge depth when it was about 2.9, 2.6, and 2.5, respectively. The experimental findings demonstrated that the frequency cancellation technique and wavelet packet decomposition-based chattering frequency monitoring approach can precisely track the chattering state of the system.