Curiosity-driven learning is foundational to human cognition. By enabling humans to autonomously decide when and what to learn, curiosity has been argued to be crucial for self-organizing temporally extended learning curricula. However, the mechanisms driving people to set intrinsic goals, when they are free to explore multiple learning activities, are still poorly understood. Computational theories propose different heuristics, including competence measures (e.g. percent correct, or PC) and learning progress (LP), that could be used as intrinsic utility functions to efficiently organize exploration. Such intrinsic utilities constitute computationally cheap but smart heuristics to prevent people from laboring in vain on random activities, while still motivating them to self-challenge on difficult learnable activities. Here, we provide empirical evidence for these ideas by means of a novel experimental paradigm and computational modeling. We show that while humans rely on competence information to avoid easy tasks, models that include an LP component provide the best fit to task selection data. These results provide a new bridge between research on artificial and biological curiosity, reveal strategies that are used by humans but have not been considered in computational research, and provide new tools for probing how humans become intrinsically motivated to learn and acquire interests and skills on extended time scales.