We give a classification of all non-symplectic automorphisms of prime order p acting on irreducible holomorphic symplectic fourfolds deformation equivalent to the Hilbert scheme of two points on a K3 surface, for p = 2, 3 and 7 ≤ p ≤ 19. Our classification relates the isometry classes of two natural lattices associated to the action of the automorphism on the second cohomology group with integer coefficients with some invariants of the fixed locus and we provide explicit examples. As an application, we find new examples of non-natural non-symplectic automorphisms.