The mass adoption of digital technologies continues to generate psychological debate on how they impact people and society. For example, associations have regularly observed between technology use and a variety of negative outcomes including depression and anxiety. However, large, pre-registered studies have failed to replicate these findings. Regardless of direction, the majority of designs rely on self-reported ‘usage’ scales. Given their importance for research integrity, here we consider what these scales are measuring. Across two studies, we observe that many scales align with a single, identical construct despite claims they capture something unique. We then demonstrate overlap between these scales and mental health by examining latent relationships. Our results suggest that psychologists need to critically consider how they proceed both methodologically and conceptually when developing psychometric scales in this domain if research findings are to ever be drawn together into a coherent body of knowledge.