Recognizing the desired herb among thousands of herbs is an exhausting and time-consuming practice. Hence, herbs identification via a vision system is beneficial since the pharmacist and botanic need not to collect them through traditional ways. Thus, this paper proposed an efficient and automatic classification system to recognize Malaysian herbs that would be used in medical or cooking areas. As per the authors' knowledge, there is no evidence for similar studies on medical herbs in Malaysia. In the proposed system, we have investigated different classifiers to build an efficient classifier; then, the classifier was integrated with a mobile app to ease the real-time classification. The proposed system employed two classifiers, namely Support Vector Machine (SVM) and Deep Learning Neural Network (DLNN). The two models have been tested on our own dataset, which contains 1000 leaves. The experimental results showed that SVM achieved 74.63% recognition accuracy and DLNN achieved 93% recognition accuracy for both of the experimental model and the developed mobile app. Furthermore, the processing time was 4 seconds for SVM and 5 seconds for DLNN classifier, while the processing time using the mobile app was 2 seconds only.