Terrain classification is a hot topic in polarimetric synthetic aperture radar (PolSAR) image interpretation that aims at assigning a label to every pixel and forms a label matrix for a PolSAR image. From the perspective of human interpretation, classification is not in terms of pixels, but decomposed perceptual groups and structures. Therefore, a new perspective of label matrix completion is proposed to treat the classification task as a label matrix inversion process. Firstly, a matrix completion framework is built to avoid processing the large-scale PolSAR image data for traditional feature and classifier learning. Secondly, a light network is used to obtain the known labels for the complete task by uniform down-sampling of the entire image that aims to keep the shape of regions in a PolSAR image and reduce the computational complexity. Finally, the zeroth-and first-order label information is proposed as the prior distribution for label matrix completion to keep the structure and texture. The experiments are tested on real PolSAR images, which demonstrates that the proposed method can realize excellent classification results with less computation time and labeled pixels. Classification results with different down-sampling rates in the light network also prove the robustness of this method.