The effective dynamics of a colloidal particle immersed in a complex medium is often described in terms of an overdamped linear Langevin equation for its velocity with a memory kernel which determines the effective (time-dependent) friction and the correlations of fluctuations. Recently, it has been shown in experiments and numerical simulations that this memory may depend on the possible optical confinement the particle is subject to, suggesting that this description does not capture faithfully the actual dynamics of the colloid, even at equilibrium. Here, we propose a different approach in which we model the medium as a Gaussian field linearly coupled to the colloid. The resulting effective evolution equation of the colloidal particle features a non-linear memory term which extends previous models and which explains qualitatively the experimental and numerical evidence in the presence of confinement. This non-linear term is related to the correlations of the effective noise via a novel fluctuation-dissipation relation which we derive.