Tongue diagnosis is an important way of diagnosing the human health in Indian ayurvedic medicine (IAM), which helps to identify different diseases through tongue image analysis. Traditionally, this diagnosis system has been done by ayurvedic experts to recognize the type of health issue by observing the human tongue. But the recent advancements in machine learning algorithms in medical field enabled the researchers to implement a computer aided diagnosis system for ayurvedic medical treatment. However, the existing tongue diagnosis systems are suffering with poor classification performance due to the variations in tongue appearance such as color, shape, coating, and texture properties. Therefore, this article focuses on computer aided tongue diagnosis system (CATDS) for disease predication through tongue image analysis, here after denoted as CATDSNet. Initially, fast nonlocal mean (FNLM) filtering is applied on test image to perform the preprocessing of given tongue dataset. Next, color features are extracted from the pre-processed tongue images using colour moments. In addition, grey level cooccurrence matrix (GLCM) is used to extract the texture features information. Then, this extracted color, and texture features are used to train the proposed CATDSNet using hybrid extreme learning machine (HELM) classifier for classifying various diseases such as healthy, appendicitis, bronchitis, gastritis, heart disease, and pancreatitis. The obtained simulation results on tongue image dataset disclose the superiority of proposed CATDSNet model as compared to state-of-the art approaches such as random forest, support vector machine (SVM), and SVM-based recursive feature elimination (SVM-RFE) with a classification accuracy of 92.3%, precision of 92.39%, recall of 92.06% and F1-score of 92.02%.