We revisit the dynamical action, developed in earlier studies [1][2][3], of the gravitational analog of Yang-Mills field, called the diffeomorphism field. We show an inconsistency in the construction of this action and solve it by a modification.The modified action becomes structurally similar to the Yang-Mills action. Then we explain the problems that arise from interpreting the diffemorphism field as a tensor. We leave out this interpretation, thereby also covariantization as a method to reach diffeomorphism invariance. By introducing corrections that involve affine connection coefficients we recover full diffeomorphism invariance in interactions of the diffeomorphism field with other fields. We also show that this approach maintains the spatial diffeomorphism invariance of the theory itself.