Crop phenology has a vital role in sustainable agriculture, facilitating farmers to make informed decisions throughout the crop-growing season. The traditional method of phenological detection relies on vegetation index calculations and time-series data, which can be extremely costly and difficult to obtain. In contrast, deep learning algorithms can estimate phenological stages directly from images, overcoming Vegetative Index (VI)-based limitations. Unmanned Aerial Vehicles (UAVs) offer high spatial and temporal resolution images at low cost, making them suitable for frequent field monitoring. This study focuses on the classification of rice seedling growth stages using deep learning techniques from images captured by UAVs. The proposed PaddyTrimNet model incorporates neural network pruning to classify paddy growth stages efficiently based on the BBCH (Biologische Bundesanstalt, Bundessortenamt und Chemische Industrie) scale. It focuses on the BBCH11, BBCH12, and BBCH13 using UAV images. PaddyTrimNet is an architecture based on ResNet50 modified specifically to classify rice development stages, incorporating separable convolutional layers to reduce parameters. The model is pruned using the Layer-wise Relevance Propagation method to enhance efficiency without compromising performance. It has demonstrated superior performance in paddy growth stage classification, achieving an accuracy of 96.97% while utilizing only 48.18 MFLOPS. It surpasses the existing pretrained deep learning classification models in terms of both accuracy and computational efficiency. This study contributes to precision agriculture and sustainable farming practices by leveraging deep learning and UAV imagery.