A discrete group G is called rigidly symmetric if the projective tensor product between the convolution algebra ℓ 1 (G) and any C * -algebra A is symmetric. We show that in each topologically graded C * -algebra over a rigidly symmetric group there is a ℓ 1 -type symmetric Banach * -algebra, which is inverse closed in the C * -algebra. This includes new general classes, as algebras admitting dual actions and partial crossed products. Results including convolution dominated kernels, inverse closedness with respect with ideals or weighted versions of the ℓ 1 -decay are included. Various concrete examples are presented.