Impulse and Gaussian are the two most common types of noise that affect digital images due to imperfections in the imaging process, compression, storage and communication. The conventional filtering approaches, however, reduce the image quality in terms of sharpness and resolution while suppressing the effects of noise. In this work, a machine learning-based filtering structure has been proposed preserves the image quality while effectively removing the noise. Specifically, a support vector machine classifier is employed to detect the type of noise affecting each pixel to select an appropriate filter. The choice of filters includes Median and Bilateral filters of different kernel sizes. The classifier is trained using example images with known noise parameters. The proposed filtering structure has been shown to perform better than the conventional approaches in terms of image quality metrics. Moreover, the design has been implemented as a hardware accelerator on an FPGA device using high-level synthesis tools.