Knee osteoarthritis (OA) is a prevalent condition that typically affects elderly individuals. It occurs due to the gradual deterioration of hyaline cartilage located between knee joints. If it not treated in earlier stage, it causes knee replacement so the knee OA early diagnosis is essential for better treatment. The knee OA diagnosis includes knee X-ray images and classifying them using kellgren-lawrence (KL) grading system. This paper proposed the application of pre-trained models named as skip connection based ResNet101 for knee OA images from osteoarthritis initiative (OAI) dataset. The skip connection based ResNet101 is utilized to overcome the vanishing gradient problems. Moreover, this skip connection allows learning of the recognized functions. This enables the ResNet101 model have performance high-level layers compared to the low-level layers. Due to these features the skip connection based ResNet101 model is utilized in this paper. The two various classifications are performed in this paper such as binary and severity. The binary classification determines whether knee osteoarthritis (OA) is absent or present, which also categorizing the severity of KOA into three grades. The experiments are conducted on three various datasets named dataset I, II & III with five, two and three grades of knee OA images. The obtained result shows that the skip connection based ResNet101 model achieves better accuracy of 79.94%, 84.96% and 95.86% in the above-mentioned three datasets when compared to ResNet101 and InceptionNetV2 models.