We investigated Norwegian children’s (n = 262) development in arithmetic fluency from first to third grade. Children’s arithmetic fluency was measured at four time points, domain-specific (i.e., symbolic numerical magnitude processing and verbal counting) and domain-general skills (i.e., working memory, rapid naming, non-verbal reasoning, and sustained attention) once in the first grade. Based on a series of growth mixture models, one developmental trajectory described the data best. Multigroup latent growth curve models showed that girls and boys developed similarly in their arithmetic fluency over time. Symbolic numerical magnitude processing and counting skills predicted both the initial level and the growth in arithmetic fluency, and working memory only the initial level, similarly for boys and girls. Mother’s educational level predicted the initial level of boys’ arithmetic fluency and rapid naming predicted the growth for girls. Our findings emphasize the role of domain-specific skills in the development of arithmetic fluency.