Monitoring the level of biothiols in organisms would be beneficial for health inspections. Recently, 3-(2′-nitro vinyl)-4-phenylselenyl coumarin as a fluorescent probe for distinguishing the detection of the small-molecule biothiols cysteine/homocysteine (Cys/Hcy) and glutathione (GSH) was developed. By introducing 4-phenyselenium as the active site, the probe CouSeNO2/CouSNO2 was capable of detecting Cys/Hcy and GSH in dual fluorescence channels. Theoretical insights into the fluorescence sensing mechanism of the probe were provided in this work. The details of the electron excitation process in the probe and sensing products under optical excitation and the fluorescent character were analyzed using the quantum mechanical method. All these theoretical results would provide insight and pave the way for the molecular design of fluorescent probes for the detection of biothiols.