Purpose Caries status detection and classification in its early stages is an essential process for a precise, less pain caries removal procedure. The aim of this work was to reveal and classify different caries degree status via an optical imaging system interfaced with Pc, and controlled by an inspection algorithm using DSP 6 software. Methods This work presents a 635-nm He-Ne laser-tissue interaction mechanism for human teeth characterization. Spectroscopic measurement for both sound and caries teeth lesion area was used to estimate the averaged optical properties. One-layer Monte Carlo model is implemented for sound, and caries lesion separately to optimize the source detector position with respect to the sample. Captured images were further processed by the designed inspection algorithm based on a Hilbert transform edge detection technique. Results The proposed optical imaging system was able to discriminate between the examined 12 tooth samples into sound and caries according to optical properties calculated from the spectroscopy measurements, where the reflectance, transmittance, and absorbance of the sound tooth was T n = 68%, R n = 23%, and A n = 0.47 o.d., respectively. In contrast, for caries values, T c reached 30%, R c = 5.2%, and A c = 1.6 o.d. Caries degrees inside cavity lesion were classified by the designed inspection algorithm into sound, moderate, and severe degree. Conclusions This study reveals the ability of the real-time, non-invasive custom optical imaging system to differentiate between sound and caries lesion teeth, in addition to classifying caries degree from images processed by the 2-D Hilbert transform inspection algorithm.