This study aims to investigate the core cognitive factors that affect reading and math performance of children of the grades 1–6 in Xi’an, Shaanxi Province, China, as well as the differences between children with dyslexia and mathematical disabilities (MD). Therefore, this study mainly evaluated the Cattell Horn Carroll (CHC) cognitive factors for 427 Chinese children and explored the core cognitive factors that affect Chinese children’s reading and math performance. Students with dyslexia (n = 34), students with mathematics learning disabilities (n = 34), and 34 normal children were randomly selected as the control group. In order to explore the differences in cognitive development, we analyzed the differences among the three groups (Dyslexia, mathematical learning disabilities (MD), and normal children). The results revealed the following: (1) almost all cognitive ability factors in this study are significantly related to students’ reading and mathematical achievements. (2) the core cognitive factors for predicting Chinese dyslexia students are crystallized intelligence, auditory processing and working memory. Executive function, spatial relationship and working memory are the core cognitive factors to predict Chinese children’s mathematical achievements. (3) in addition, there are differences in cognitive deficits between disabled Chinese children in reading and math, among which those with reading deficits have extensive auditory processing deficits; while children with mathematic deficits have worse executive function. Recommendations were made based on these findings.