The human abdominal region is very heterogeneous and stratified with subcutaneous adipose tissue (SAT) being one of the primary layers. Monitoring this tissue is crucial for diagnostic purposes and to estimate the effects of interventions like caloric restriction or bariatric surgery. However, the layered nature of the abdomen poses a major problem in monitoring the SAT in a non-invasive way by diffuse optics. In this work, we examine the possibility of using multi-distance broadband time domain diffuse optical spectroscopy to assess the human abdomen non-invasively. Broadband absorption and reduced scattering spectra from 600 to 1100 nm were acquired at 1, 2 and 3 cm source-detector distances on ten healthy adult male volunteers, and then analyzed using a homogeneous model as an initial step to understand the origin of the detected signal and how tissue should be modeled to derive quantitative information. The results exhibit a clear influence of the layered nature on the estimated optical properties. Clearly, the underlying muscle makes a relevant contribution in the spectra measured at the largest source-detector distance for thinner subjects related to blood and water absorption. More unexpectedly, also the thin superficial skin layer yields a direct contamination, leading to higher water content and steeper reduced scattering spectra at the shortest distance, as confirmed also by simulations. In conclusion, provided that data analysis properly accounts for the complex tissue structure, diffuse optics may offer great potential for the continuous non-invasive monitoring of abdominal fat.