Railway track faults may lead to railway accidents and cause human and financial loss. Spatial, temporal, and weather elements, and wear and tear, lead to ballast, loose nuts, misalignment, and cracks leading to accidents. Manual inspection of such defects is time-consuming and prone to errors. Automatic inspection provides a fast, reliable, and unbiased solution. However, highly accurate fault detection is challenging due to the lack of public datasets, noisy data, inefficient models, etc. To obtain better performance, this study presents a novel approach that relies on mel frequency cepstral coefficient features from acoustic data. The primary objective of this study is to increase fault detection performance. As well as designing an ensemble model, we utilize selective features using chi-square(chi2) that have high importance with respect to the target class. Extensive experiments were carried out to analyze the efficiency of the proposed approach. The experimental results suggest that using 60 features, 40 original features, and 20 chi2 features produces optimal results both regarding accuracy and computational complexity. A mean accuracy score of 0.99 was obtained using the proposed approach with machine learning models using the collected data. Moreover, this performance was significantly better than that of existing approaches; however, the performance of models may vary in real-world settings.