Computed tomography (CT) scanning is a fast and painless procedure that can capture clear imaging information beneath the abdomen and is widely used to help diagnose and monitor disease progress. The pelvic lymph node is a key indicator of colorectal cancer metastasis. In the traditional process, an experienced radiologist must read all the CT scanning images slice by slice to track the lymph nodes for future diagnosis. However, this process is time‐consuming, exhausting, and subjective due to the complex pelvic structure, numerous blood vessels, and small lymph nodes. Therefore, automated methods are desirable to make this process easier. Currently, the available open‐source CTLNDataset only contains large lymph nodes. Consequently, a new data set called PLNDataset, which is dedicated to lymph nodes within the pelvis, is constructed to solve this issue. A two‐level annotation calibration method is proposed to guarantee the quality and correctness of pelvic lymph node annotation. Moreover, a novel system composed of a keyframe localization network and a lymph node detection network is proposed to detect pelvic lymph nodes in CT scanning images. The proposed method makes full use of two kinds of prior knowledge: spatial prior knowledge for keyframe localization and anchor prior knowledge for lymph node detection. A series of experiments are carried out to evaluate the proposed method, including ablation experiments, comparing other state‐of‐the‐art methods, and visualization of results. The experimental results demonstrate that our proposed method outperforms other methods on PLNDataset and CTLNDataset. This system is expected to be applied in future clinical practice.