The concept of the polaron in condensed matter physics has been extended to the Rabi model, where polarons resulting from the coupling between a two-level system and single-mode photons represent two oppositely displaced oscillators. Interestingly, tunneling between these two displaced oscillators can induce an anti-polaron, which has not been systematically explored in the literature, especially in the presence of an asymmetric term. In this paper, we present a systematic analysis of the competition between the polaron and anti-polaron under the interplay of the coupling strength and the asymmetric term. While intuitively the anti-polaron should be secondary owing to its higher potential energy, we find that, under certain conditions, the minor anti-polaron may gain a reversal in the weight over the major polaron. If the asymmetric amplitude ǫ is smaller than the harmonic frequency ω, such an overweighted anti-polaron can occur beyond a critical value of the coupling strength g; if ǫ is larger, the anti-polaron can even be always overweighted at any g. We propose that the explicit occurrence of the overweighted anti-polaron can be monitored by a displacement transition from negative to positive values. This displacement is an experimentally accessible observable, which can be measured by quantum optical methods, such as balanced Homodyne detection.