Accurate diagnosis of various lesions in the formation stage of gastric cancer is an important problem for doctors. Automatic diagnosis tools based on deep learning can help doctors improve the accuracy of gastric lesion diagnosis. Most of the existing deep learning-based methods have been used to detect a limited number of lesions in the formation stage of gastric cancer, and the classification accuracy needs to be improved. To this end, this study proposed an attention mechanism feature fusion deep learning model with only 14 million (M) parameters. Based on that model, the automatic classification of a wide range of lesions covering the stage of gastric cancer formation was investigated, including non-neoplasm(including gastritis and intestinal metaplasia), low-grade intraepithelial neoplasia, and early gastric cancer (including high-grade intraepithelial neoplasia and early gastric cancer). 4455 magnification endoscopy with narrow-band imaging(ME-NBI) images from 1188 patients were collected to train and test the proposed method. The results of the test dataset showed that compared with the advanced gastric lesions classification method with the best performance (overall accuracy = 94.3%, parameters = 23.9 M), the proposed method achieved both higher overall accuracy and a relatively lightweight model (overall accuracy =95.6%, parameter = 14 M). The accuracy, sensitivity, and specificity of low-grade intraepithelial neoplasia were 94.5%, 93.0%, and 96.5%, respectively, achieving state-of-the-art classification performance. In conclusion, our method has demonstrated its potential in diagnosing various lesions at the stage of gastric cancer formation.