The color potato has the function of both a food and vegetable. The color potato not only contains various amino acids and trace elements needed by the human body but also contains anthocyanins. Anthocyanins have many functions, such as antioxidation, inflammation inhibition, vision improvement, and cancer prevention, so colored potatoes are deeply loved by consumers and have good market prospects. However, at present, the detection of anthocyanin content in color potatoes mainly depends on chemical methods, which are time-consuming and laborious, so it is necessary to study a fast and accurate detection method. In this study, microscopic hyperspectral equipment was used to collect the spectral information of the outer skin and inner skin of potatoes. The original spectrum, pretreatment spectrum, and characteristic spectrum variables of the outer skin and inner skin were predicted by the convolution neural network (CNN) algorithm and partial least squares regression (PLS) algorithm, respectively, and the performance of the model was evaluated by the prediction set correlation coefficient (Rp), prediction set root mean square error (RMSEP), correction set correlation coefficient (Rc), correction set root mean square error (RMSEC), and residual prediction deviation (RPD). The results revealed that the inner skin Raw + CNN model constructed under raw spectral data is optimal with Rc = 0.9508, RMSEC = 0.0374%, Rp = 0.9461, RMSEP = 0.2361% and RPD = 4.4933. The inner skin Savitzky-Golay (SG) + Detrend (DET) + CNN model constructed from pre-processed spectral data is optimal with Rc = 0.9499, RMSEC = 0.0359%, Rp = 0.9439, RMSEP = 0.2384%, RPD = 4.6516. The inner skin DET + competitive adaptive reweighted sampling (CARS) +CNN model constructed from the feature-based spectral data was optimal with Rc = 0.9527, RMSEC = 0.0708%, Rp = 0.9457, RMSEP = 0.2711%, and RPD = 4.1623. It can be seen that the Rp, RMSEP, Rc, RMSEC, and RPD values for modeling the spectral information of the inner skin were higher than those of the outer skin under the three different spectral data. The prediction accuracy of the model built by the CNN algorithm was better than the conventional algorithm PLS, the application of the CNN algorithm in inner skin can achieve accurate prediction of anthocyanin content in potato.