Visible-light optical coherence tomography (vis-OCT) enabled new spectroscopic applications, such as retinal oximetry, as a result of increased optical absorption and scattering contacts in biological tissue and improved axial resolution. Besides extracting tissue properties from back-scattered light, spectroscopic analyses must consider spectral alterations induced by image reconstruction itself. We investigated an intrinsic spectral bias in the background noise floor, which is hereby referred to as the spectrally-dependent background (SDBG). We developed an analytical model to predict the SDBG-induced bias and validated this model using numerically simulated and experimentally acquired data. We found that SDBG systemically altered the measured spectra of blood in human retinal vessels in vis-OCT, as compared to literature data. We provided solutions to quantify and compensate for SDBG in retinal oximetry. This work is particularly significant for clinical applications of vis-OCT.