Polycystic ovary syndrome (PCOS) is a disorder in the female ovary caused because of reproductive age group hormonal changes. PCOS is a different follicle that is formed in the ovary and is termed an endocrine disorder. This disorder’s effects are often linked with clinical symptoms such as arteries, acne, hirsutism, diabetes, cardiovascular disease, and chronic infertility. It is mainly associated with type 2 diabetes, obesity with high cholesterol. This must be diagnosed at an earlier stage for avoiding other related diseases. To ensure infertility, various kinds of ovulatory failures must be significantly diagnosed and recognized. The physicians manually determine the PCOS using ultrasound images, but it is inefficient to declare whether it is a simple cyst, PCOS, or cancer cyst. This manual detection is prone to trying errors. In this paper, PCOS detection is performed through a series of processes such as preprocessing, segmentation, feature selection, and classification. The speckle noise is removed in preprocessing, and the images are enhanced for further processing. The proposed improved adaptive K-means with reptile search algorithm (IAKmeans-RSA) has been utilized for cyst segmentation and follicles recognition. The relevant features from the segmented images are extracted using a convolutional neural network (CNN). Finally, the classification is performed using the Deep Neural Network (DNN) approach. The proposed system efficiently diagnosed PCOS through cyst detection from the input images. The algorithm’s efficiency compared with existing methods shows that the proposed model is superior in segmenting and diagnosing PCOS.