Ripeness estimation of fruits is an essential process that impact the quality of fruits and its marketing. Nearly 30% to 35% get wasted from the harvested fruits due to lack of skilled workers in classification and fruit grading. Although it can be executed by human assessment, it is time consuming, costlier and error prone. Lot of research is carried to automate the quality assessment of fruits. Several hyper-parameters have been considered which have liven up by providing robust convolutional neural network (CNN). This paper has focused on image resizer stochastic gradient descent (SGD) algorithm for computing the loss. It updates the parameter by concentrating channels with respect to red, green, and blue (RGB) to identify and classify the images as ripen and rotten. The real time dataset (6702 images) of oranges, papaya and banana is collected. Using SGD optimizer, learning rate of 0.01 and nearest neighbor interpolation algorithm as resizer, the proposed model has achieved accuracy rate of 96.56% after 38 epochs in classifying the fruits as ripen and rotten. It is also observed that it is possible to use small dataset on visual geometry group with 16 layer (VGG) with the above specification and good accuracy rate can be achieved.