Polyps are a group of cells growing on the inner surface of the colon. Over time, some polyps can lead to colon cancer, which is often fatal if found in its later stages. Colon cancer can be prevented if the polyps are identified and removed in their early stages. Colonoscopy is a very effective screening method to remove polyps and it largely prevents colon cancer. However, some polyps may not be detected during a colonoscopy due to human error. Over the past two decades, many studies have been conducted on computer‐aided detection to reduce the miss rate of polyps. This study consists of two distinct parts, the detection of frames containing polyps and polyp segmentation. In the first section, a new convolutional neural network based on the VGG network is proposed. The proposed network has an accuracy of 86% on a newly collected dataset. In the polyp segmentation section, a fully convolutional network and an effective post‐processing algorithm are presented. An evaluation of the proposed polyp segmentation system on the ETIS‐LARIB database achieves an overall 82.00% F2 score, which outperforms the methods that participated in the sub‐challenge of MICCAI.