We give a criterion for the rigidity of the action of a group of affine transformations of a homogeneous space of a real Lie group. Let G be a real Lie group, Λ a lattice in G, and Γ a subgroup of the affine group Aff(G) stabilizing Λ. Then the action of Γ on G/Λ has the rigidity property in the sense of S. Popa [Pop06], if and only if the induced action of Γ on P(g) admits no Γ-invariant probability measure, where g is the Lie algebra of G. This generalizes results of M. Burger [Bur91], and A. Ioana and Y. Shalom [IS13]. As an application, we establish rigidity for the action of a class of groups acting by automorphisms on nilmanifolds associated to step 2 nilpotent Lie groups.