The extraction of power line plays a key role in power line inspection by Unmanned Aerial Vehicles (UAVs). While it is challenging to extract power lines in aerial images because of the weak targets and the complex background. In this paper, a novel power line extraction method is proposed. First of all, we create a line segment candidate pool which contains power line segments and large amount of other line segments. Secondly, we construct the irregular graph model with these line segments as nodes. Then a novel object-based Markov random field with anisotropic weighted penalty (OMRF-AWP) method is proposed. It defines a new neighborhood system based on the irregular graph model and builds a new potential function by considering the region angle information. With the OMRF-AWP method, we can distinguish between the power line segments and other line segments. Finally, an envelope-based piecewise fitting (EPF) method is proposed to fit the power lines. Experimental results show that the proposed method has good performance in multiple scenes with complex background. INDEX TERMS Power line extraction, object-based Markov random field (OMRF), anisotropic weighted penalty, irregular graph model, potential function.