The Cu-filling process in through-silicon via (TSV-Cu) is a key technology for chip stacking and three-dimensional vertical packaging. During this process, defects resulting from chemical–mechanical planarization (CMP) and annealing severely affect the reliability of the chips. Traditional methods of defect characterization are destructive and cumbersome. In this study, a new defect inspection method was developed using Mueller matrix spectroscopic ellipsometry. TSV-Cu with a 3-µm-diameter and 8-µm-deep Cu-filling showed three typical types of defects: over-dishing (defect-OD), protrusion (defect-P), and defect-free defects. The process window for each defect was 13 nm. First, the three typical defects caused by CMP and annealing were investigated. With single-channel deep learning and a Mueller matrix element (MME), the TSV-Cu defect types could be distinguished with an accuracy rate of 99.94%. Next, seven effective MMEs were used as seven channels in the artificial neural network to quantify the height variation in the Cu-filling in the z-direction. The accuracy rate was 98.92% after training, and the recognition accuracy reached 1 nm. The proposed approach rapidly and nondestructively evaluates the annealing bonding and CMP processes, which can improve the reliability of high-density integration.