Numerical processing has been demonstrated to be subserved typically by the brain regions around the bilateral intraparietal sulcus (IPS). The goal of the current study was to investigate whether the processing of mathematical terms shared the same brain regions with numerical processing. Healthy adult participants performed semantic distance judgment tasks on five types of materials, including geometric terms, algebraic terms, linguistic terms, words for tools and other common objects, and Arabic numbers. Brain activation was measured with functional magnetic resonance imaging (fMRI). The results showed that geometric terms had greater activation than algebraic terms, linguistic terms and tool words in the horizontal IPS, but algebraic terms did not have greater activation than linguistic terms and tool words in this region. Arabic numbers showed greater activation than non-number materials (including geometric terms, algebraic terms, linguistic terms and tool words) in the bilateral IPS, right inferior frontal gyrus and bilateral middle frontal gyrus, but the non-number materials showed stronger activation in the left inferior frontal gyrus and left middle temporal gyrus. These results suggest that the brain area for the processing of numbers (the left IPS) seems to be involved in semantic processing of geometric terms, but not that of other mathematical terms such as algebraic terms. Both algebraic and geometric terms share similar brain organization with basic semantic processing in the left temporal and frontal regions.