The non-linear gravitational wave (GW) memory effect is a distinct prediction in general relativity. While the effect has been well studied for comparable mass binaries, it has mostly been overlooked for intermediate mass ratio inspirals (IMRIs). We offer a comprehensive analysis of the phenomenology and detectability of memory effects, including contributions from subdominant harmonic modes, in heavy IMRIs consisting of a stellar mass black hole and an intermediate mass black hole. When formed through hierarchical mergers, for example when a GW190521-like remnant captures a stellar mass black hole, IMRI systems have a large total mass, large spin on the primary, and possibly residual eccentricity; features that potentially raise the prospect for memory detection. We compute both the displacement and spin non-linear GW memory from the m = 0 gravitational waveforms computed within a black hole perturbation theory framework that is partially calibrated to numerical relativity waveforms. We probe the dependence of memory effects on mass ratio, spin, and eccentricity and consider the detectability of a memory signal from IMRIs using current and future GW detectors. We find that (i) while eccentricity introduces additional features in both displacement and spin memory, it does not appreciatively change the prospects of detectability, (ii) including higher modes into the memory computation can increase singal-to-noise (SNR) values by about 7% in some cases, (iii) the SNR from displacement memory dramatically increases as the spin approaches large, positive values, (iv) spin memory from heavy IMRIs would, however, be difficult to detect with future generation detectors even from highly spinning systems. Our results suggest that hierarchical binary black hole mergers may be a promising source for detecting memory and could favorably impact memory forecasts.