In recent years philosophers have used results from cognitive science to formulate epistemologies of arithmetic (e.g. Giaquinto in J Philos 98(1):5–18, 2001). Such epistemologies have, however, been criticised, e.g. by Azzouni (Talking about nothing: numbers, hallucinations and fictions, Oxford University Press, 2010), for interpreting the capacities found by cognitive science in an overly numerical way. I offer an alternative framework for the way these psychological processes can be combined, forming the basis for an epistemology for arithmetic. The resulting framework avoids assigning numerical content to the Approximate Number System and Object Tracking System, two systems that have so far been the basis of epistemologies of arithmetic informed by cognitive science. The resulting account is, however, only a framework for an epistemology: in the final part of the paper I argue that it is compatible with both platonist and nominalist views of numbers by fitting it into an epistemology for ante rem structuralism and one for fictionalism. Unsurprisingly, cognitive science does not settle the debate between these positions in the philosophy of mathematics, but I it can be used to refine existing epistemologies and restrict our focus to the capacities that cognitive science has found to underly our mathematical knowledge.