Aimed to automate the segmentation of organs at risk (OARs) in head and neck (H&N) cancer radiotherapy, we develop a novel Prior Attention enhanced convolutional neural Network (PANet) based Stepwise Refinement Segmentation Framework (SRSF) on full-size computed tomography (CT) images. The SRSF is built with a multiscale segmentation concept, in which OARs are segmented from coarse to fine. PANet is a pyramidal architecture with elements of inception block and prior attention. In this study, the developed PANet based SRSF is applied for OARs segmentation in H&N radiotherapy. 139 CT series and manually delineated contours of twenty-two OARs by experienced oncologists are collected from 139 H&N patients for training and evaluating the proposed PANet based SRSF. The mean testing Dice similarity coefficients (DSC) on 39 CT series range from 76.1±8.3% (left middle ear) to 91.9±1.4% (right mandible) for large volume OARs(mean volume>1cc) while the corresponding ranges are 63.4±12.3%(chiasm) to 81.0±14.1% (right lens) for small and challenging OARs(mean volume1cc). Furthermore, the proposed method also achieved superior segmentations over reference methods on the MICCAI 2015 H&N dataset with mean DSC of 95.6±0.7%, 81.3±4.0%, 77.6±4.5%, 77.5±4.6%, and 69.2±7.6%, on the mandible, left submandibular, left and right optical nerve, and chiasm, respectively. The accurate segmentation of OARs is obtained on both the self-collected testing data and public testing dataset, which implies that the proposed method can be used as a practicable and efficient tool for automated OARs contouring in the H&N cancer radiotherapy.