Cancer is a disease involving unusual cell growth likely to spread to other parts of the body. According to WHO 2020 report, colorectal malignancy is the globally accepted second leading cause of cancer related deaths. Colorectal malignancy arises when malignant cells often called polyp, grow inside the tissues of the colon or rectum of the large intestine. Colonoscopy, CT scan, Histopathological analysis are some manual approaches of malignancy detection that are time consuming and lead to diagnostic errors. Supervised CNN data model requires a large number of labeled training samples to learn parameters from images. In this study we propose an expert system that can detect the colorectal malignancy and identify the exact polyp area from complex images. In this approach an unsupervised Deep Belief Network (DBN) is applied for effective feature extraction and classification of images. The classified image output of DBN is utilized by Polyp Detector. Residual network and feature extractor components of Polyp Detector helps polyp inspector in pixel wise learning. Two stage polyp network (PLPNet) is a R-CNN architecture with two stage advantage. The first stage is the extension of R-CNN to detect the polyp lesion area through a location box also called Polyp Inspector. Second Stage performs polyp segmentation. Polyp Inspector transfers the learned semantics to the polyp segmentation stage. It helps to enhance the ability to detect polyp with improved accuracy and guide the learning process. Skip schemes enrich the feature scale. Publicly available CVC-Clinical DB and CVC Colon DB datasets are used for experiment purposes to achieve a better prediction capability for clinical practices.