We study numerically joint mixing of salt and colloids by chaotic advection and how salt inhomogeneities accelerate or delay colloid mixing by inducing a velocity drift V(dp) between colloids and fluid particles as proposed in recent experiments [J. Deseigne et al., Soft Matter 10, 4795 (2014)]. We demonstrate that because the drift velocity is no longer divergence free, small variations to the total velocity field drastically affect the evolution of colloid variance σ(2) = 〈C(2)〉-〈C〉(2). A consequence is that mixing strongly depends on the mutual coherence between colloid and salt concentration fields, the short time evolution of scalar variance being governed by a new variance production term P = -〈C(2)∇ · V(dp)〉/2 when scalar gradients are not developed yet so that dissipation is weak. Depending on initial conditions, mixing is then delayed or enhanced, and it is possible to find examples for which the two regimes (fast mixing followed by slow mixing) are observed consecutively when the variance source term reverses its sign. This is indeed the case for localized patches modeled as Gaussian concentration profiles.