Potholes are one type of pavement surface distresses whose assessment is essential for developing road network maintenance strategies. Existing methods for automatic pothole detection either rely on expensive and high-maintenance equipment or could not segment the pothole accurately. In this paper, an asphalt pavement pothole detection and segmentation method based on energy field is put forward. The proposed method mainly includes two processes. Firstly, the wavelet energy field of the pavement image is constructed to detect the pothole by morphological processing and geometric criterions. Secondly, the detected pothole is segmented by Markov random field model and the pothole edge is extracted accurately. This methodology has been implemented in a MATLAB prototype, trained, and tested on 120 pavement images. The results show that it can effectively distinguish potholes from cracks, patches, greasy dirt, shadows, and manhole covers and accurately segment the pothole. For pothole detection, the method reaches an overall accuracy of 86.7%, with 83.3% precision and 87.5% recall. For pothole segmentation, the overlap degree between the extracted pothole region and the original pothole region is mostly more than 85%, which accounts for 88.6% of the total detected pavement pothole images.