Skills and understanding of operations with negative numbers, which are typically taught in middle school, are crucial aspects of numerical competence necessary for all subsequent mathematics. To more swiftly and coherently develop the field's understanding of how to foster this critical competence, we need shared measures that allow us to compare results across studies with diverse populations and theoretical perspectives. Yet, to date no validated instrument exists to assess all four primary operations (addition, subtraction, multiplication and division). Thus, we conducted a Rasch analysis of the Integer Test of Primary Operations (ITPO) with 187 middle school students to provide a valid and reliable assessment with good person and item fit. The implications of this study are numerous for multiple stakeholders including scholars, test and textbook developers, as well as teachers. First, we validated three forms of the ITPO to foster future longitudinal studies of how integer arithmetic knowledge is maintained or decays as well as how such knowledge might be related to success in STEM disciplines. Second, our analysis provides trustworthy insights about relative difficulty of integer problem structures because regardless of test form similar problem structures loaded together. For instance, sums of additive inverses were the easiest structure, whereas division by -1 was more difficult than multiplying or dividing by any other integer. We discuss each of these and other findings that have practical implications for learning and teaching integers. Third, for broader mathematics assessments in which minimal items can be included to measure integer knowledge, this study informs which items would serve the intended assessment purpose. Finally, we provide the three forms as an appendix in printable formats to ensure these validated tests are practical to implement for teachers as well as scholars.