Using active tumor-targeting nanoparticles, fluorescence imaging can provide highly sensitive and specific tumor detection, and precisely guide radiation in translational radiotherapy study. However, the inevitable presence of non-specific nanoparticle uptake throughout the body can result in high levels of heterogeneous background fluorescence, which limits the detection sensitivity of fluorescence imaging and further complicates the early detection of small cancers. In this study, background fluorescence emanating from the baseline fluorophores was estimated from the distribution of excitation light transmitting through tissues, by using linear mean square error estimation. An adaptive masked-based background subtraction strategy was then implemented to selectively refine the background fluorescence subtraction. First, an in vivo experiment was performed on a mouse intratumorally injected with passively targeted fluorescent nanoparticles, to validate the reliability and robustness of the proposed method in a stringent situation wherein the target fluorescence was overlapped with the strong background. Then, we conducted in vivo studies on 10 mice which were inoculated with orthotopic breast tumors and intravenously injected with actively targeted fluorescent nanoparticles. Results demonstrated that active targeting combined with the proposed background subtraction method synergistically increased the accuracy of fluorescence molecular imaging, affording sensitive tumor detection.