The accreting millisecond X-ray pulsar SAX J1808.4-3658 shows peculiar low luminosity states known as "reflares" after the end of the main outburst. During this phase the X-ray luminosity of the source varies by up to three orders of magnitude in less than 1-2 days. The lowest X-ray luminosity observed reaches a value of 10 erg s 32 1 -, only a factor of a few brighter than its typical quiescent level. We investigate the 2008 and 2005 reflaring state of SAX J1808.4-3658 to determine whether there is any evidence for a change in the accretion flow with respect to the main outburst. We perform a multiwavelength photometric and spectral study of the 2005 and 2008 reflares with data collected during an observational campaign covering the near-infrared, optical, ultra-violet and X-ray band. We find that the NIR/optical/UV emission, expected to come from the outer accretion disk, shows variations in luminosity over an order of magnitude. The corresponding X-ray luminosity variations are instead much deeper, spanning about 2-3 orders of magnitude. The X-ray spectral state observed during the reflares does not change substantially with X-ray luminosity, indicating a rather stable configuration of the accretion flow. We investigate the most likely configuration of the innermost regions of the accretion flow and we infer an accretion disk truncated at or near the co-rotation radius. We interpret these findings as due to either a strong outflow (due to a propeller effect) or a trapped disk (with limited/no outflow) in the inner regions of the accretion flow.