Social insect colonies are often considered to be highly efficient collective systems, with division of labor at the root of their ecological success. However, in many species, a large proportion of a colony's workers appear to spend their time completely inactive. The role of this inactivity for colony function remains unclear. Here, we investigate how inactivity is distributed among workers and over time in the ant Temnothorax rugatulus. We show that the level of inactivity is consistent for individual workers, but differs significantly among workers, that is, some workers effectively specialize on 'inactivity'. We also show that workers have circadian rhythms, although intra-nest tasks tend to be performed uniformly across the whole day. Differences in circadian rhythms, or workers taking turns resting (i.e., working in shifts), cannot explain the observation that some workers are consistently inactive. Using extensive individual-level data to describe the overall structure of division of labor, we show that 'inactive workers' form a group distinct from other task groups. Hierarchical clustering suggests that inactivity is the primary variable in differentiating both workers and tasks. Our results underline the importance of inactivity as a behavioral state and the need for further studies on its evolution.