The reliability of gas insulated switchgear (GIS) is very important for the safe operation of power systems. However, the research on potential faults of GIS is mainly focused on partial discharge, and the research on the intelligent detection technology of the mechanical state of GIS is very scarce. Based on the abnormal vibration signals generated by a GIS fault, a fault diagnosis method consisting of a frequency feature extraction method based on coherent function (CF) and a multi-layer classifier was developed in this paper. First, the Fourier transform was used to analyze the differences and consistency in the frequency spectrum of signals. Secondly, the frequency domain commonalities of the vibration signals were extracted by using CF, and the vibration characteristics were screened twice by using the correlation threshold and frequency threshold to further select the vibration features for diagnosis. Then, a multi-layer classifier composed of two one-class support vector machines (OCSVMs) and one support vector machine (SVM) was designed to classify the faults of GIS. Finally, the feasibility of the feature extraction method was verified by experiments, and compared with other classification methods, the stability and reliability of the proposed classifier were verified, which indicates that the fault diagnosis method promotes the development of an intelligent detection technology of the mechanical state in GIS.