An Okounkov body is a convex subset in Euclidean space associated to a big divisor on a smooth projective variety with respect to an admissible flag. In this paper, we introduce two convex bodies associated to pseudoeffective divisors, called the valuative Okounkov bodies and the limiting Okounkov bodies, and show that these convex bodies reflect the asymptotic properties of pseudoeffective divisors as in the case with big divisors. Our results extend the works of Lazarsfeld–Mustaţă and Kaveh–Khovanskii. For this purpose, we define and study special subvarieties, called the Nakayama subvarieties and the positive volume subvarieties, associated to pseudoeffective divisors.