Pointer Meter Automatic Recognition (PMAR) under outdoor environment is a challenging task. Due to the variable weather and uneven lighting factors, hand-crafted features or shallow learning techniques have low accuracy in meter recognition. In this paper, a Multitask Cascading Convolutional Neural Network (MC-CNN) is proposed to improve the accuracy of meter recognition under outdoor environment. The proposed MC-CNN used cascaded CNN, including three stages of meter detection, meter cropping and meter reading. Firstly, the YOLOV4 network is used for meter detection to quickly determine the meter location from captured images. In order to accurately cluster pointer meters prior boxes in the YOLOV4 Network, an improved K-means algorithm is presented to further enhance the detection accuracy. Then, the detected meter images are cropped out of the captured images to remove redundant backgrounds. Finally, a meter Reading Network (RNet) based on Adaptive Attention Residual Module (AARM) is proposed for reading meters from cropped images. The proposed AARM not only contains an attention mechanism to focus on essential information and diminish the useless one well, but also extracts information features from meter images adaptively. The experimental results show that the proposed MC-CNN can effectively achieve outdoor meter recognition, with high recognition accuracy and low relative error. The recognition accuracy can reach 92.6%. Compared with the outstanding methods, the average relative error is 2.5655%, reducing by about 3%. What is more, the proposed approach can obtain rich information about the type, limits, units and readings of the pointer meter and can be used when multiple pointer meters exist in one captured image simultaneously. Meanwhile, the proposed approach can significantly improve the accuracy of the recognized readings, and is also robust to the natural environments.