We investigate the fate of diffeomorphisms when the radial gauge is imposed in canonical general relativity. As shown elsewhere, the radial gauge is closely related to the observer's observables. These observables are invariant under a large subgroup of diffeomorphisms which results in their usefulness for canonical general relativity. There are, however, some diffeomorphisms, called residual diffeomorphisms, which might be 'observed' by the observer as they do not preserve her observables. The present paper is devoted to the analysis of these diffeomorphisms in the case of the spatial and spacetime radial gauges. Although the residual diffeomorphisms do not form a subgroup of all diffeomorphisms, we show that their induced action in the phase space does form a group. We find the generators of the induced transformations and compute the structure functions of the algebras they form. The obtained algebras are deformations of the algebra of the Euclidean group and the algebra of the Poincaré group in the spatial and spacetime case, respectively. In both cases the deformation depends only on the Riemann curvature tensor and in particular vanishes when the space or spacetime is flat.