In modern machining processes, chatter is an inherent phenomenon that hinders efficiency, productivity, and automation. Numerous methods have been proposed using analytical, computational, and artificial intelligence methods to detect and avoid chatter during milling. The vibration signals generated during machining are of non-stationary and non-linearity nature. Hence solely time or frequency domain analysis are not adequate methods for chatter detection. This study investigates the performance of more advanced mode decomposition methods and compares them. Three decomposition methods, namely, empirical mode decomposition (EMD), ensemble empirical mode decomposition (EEMD), and variational mode decomposition (VMD), are used to decompose and identify chatter frequency bands. After decomposition, Hilbert-Huang transform (HHT) was applied for visualization. The comparative results indicate that EEMD or VMD decomposition methods performed better than EMD for intelligent chatter detection.