We experimentally investigate the transient recoil dynamics of a colloidal probe particle in a viscoelastic fluid after the driving force acting on the probe is suddenly removed. The corresponding recoil displays two distinct timescales which are in excellent agreement with a microscopic model which considers the probe particle to be coupled to two bath particles via harmonic springs. Notably, this model exhibits two sets of eigenmodes which correspond to reciprocal and non-reciprocal force conditions and which can be experimentally confirmed in our experiments. We expect our findings to be relevant under conditions where particles are exposed to non-steady shear forces as this is encountered e.g. in microfluidic sorting devices or the intermittent motion of motile bacteria within their natural viscoelastic surrounding.