The abacus, a traditional physical calculation device, is still widely used in Asian countries. Previous behavioral work has shown that skilled abacus users perform rapid and precise mental arithmetic by manipulating a mental representation of an abacus, which is based on visual imagery. However, its neurophysiological basis remains unclear. Here, we report the case of a patient who was a good abacus user, but transiently lost her “mental abacus” and superior arithmetic performance after a stroke owing to a right hemispheric lesion including the dorsal premotor cortex (PMd) and inferior parietal lobule (IPL). Functional magnetic resonance imaging experiments were conducted 6 and 13 months after her stroke. In the mental calculation task, her brain activity was shifted from the language-related areas, including Broca’s area and the left dorsolateral prefrontal and IPLs, to the visuospatial-related brain areas including the left superior parietal lobule (SPL), according to the recovery of her arithmetic abilities. In the digit memory task, activities in the bilateral SPL, and right visual association cortex were also observed after recovery. The shift of brain activities was consistent with her subjective report that she was able to shift the calculation strategy from linguistic to visuospatial as her mental abacus became stable again. In a behavioral experiment using an interference paradigm, a visual presentation of an abacus picture, but not a human face picture, interfered with the performance of her digit memory, confirming her use of the mental abacus after recovery. This is the first case report on the impairment of the mental abacus by a brain lesion and on recovery-related brain activity. We named this rare case “abacus-based acalculia.” Together with previous neuroimaging studies, the present result suggests an important role for the PMd and parietal cortex in the superior arithmetic ability of abacus users.