Motivated by the results involving Drazin inverses of Patr\'{i}cio and Puystjens, we investigate the relevant results for pseudo Drazin invertibility and generalized Drazin invertibility in two semigroups of Banach algebras. Given a Banach algebra $\mathcal{A}$ and $e^2=e\in \mathcal{A}$, we first establish the relation between pseudo Drazin invertibility (resp., generalized Drazin invertibility) of elements in $e\mathcal{A}e$ and $e\mathcal{A}e+1-e$. Then this result leads to a remarkable behavior of pseudo Drazin invertibility (resp., generalized Drazin invertibility) between the operators in the semigroup $AA^{-}\mathscr{B}(X)AA^{-}+I_X-AA^{-}$ and the semigroup $A^{=}A\mathscr{B}(Y)A^{=}A+I_Y-A^{=}A$, where $A^{-}, A^{=}\in \mathscr{B}(Y,X)$ are inner inverses of $A\in \mathscr{B}(X,Y)$.