Worldwide, more than 40k rice varieties are existing, each with different nutritional content and quality. Identifying these has to be consistent, automated, and accurate. Considering the feature extraction process, convolution Neural Networks (CNN) are preferred over machine learning (ML) for this classification. Transfer learning approaches help to optimize the CNN model; therefore, it fits in an FPGA. Seven different CNN models were proposed to classify five rice varieties, each model differs based on the: kernel depth; the number of convolution layers (CL); the number of fully connected layers (FCL); and the number of neurons per FCL. These were analyzed considering 70% and 30% for training and testing respectively. A dataset of 15,000 images/variety with each image of resolution. This results in an Optimized Lightweight Convolutional Neural Network (OpLW-CNN) model, having a CL, followed by two FCLs. This model is further analyzed using a random set of images: 500, 5000, and 75000 to fit the model optimally. This model achieves 99%, 98.13%, and 98.14% of specificity, F1-score, and accuracy for a set of 5000 images. These metrics are approximately 1% to 2% lesser than the performance of the benchmark model, and 81.5% fewer computations. Also, this model requires less than a second to classify an image.