The research of skin lesion diseases is currently one of the hottest topics in the medical research fields, and has gained a lot of attention on the last few years. However, the existing skin lesion methods are mainly relying on conventional Convolutional Neural Network (CNN) and the performance of skin lesion recognition is far from satisfactory. Therefore, to overcome the aforementioned drawbacks of traditional methods, we propose a novel Computer-Aided Diagnosis (CAD) system, named DermoNet, based on Multi-Scale Feature Level (MSFL) blocks and Multi-Level Feature Fusion (MLFF). Further, the DermoNet approach yields a significant enhancement in terms of dealing with the challenge of small training data sizes in the dermoscopic domain and avoiding high similarity between classes and overfitting issue. Extensive experiments are conducted on the public dermoscopic dataset, and the results demonstrate that DermoNet outperforms the state-of-the-art approaches. Hence, DermoNet can achieve an excellent diagnostic efficiency in the auxiliary diagnosis of skin lesions.