Background Current findings support the hypothesis that the exposure of dioxin-like polychlorinated biphenyls (dl-PCBs) has adverse cognitive effects even at levels that are generally considered to pose low or no risk. However, the effect of non-dioxin-like PCBs (ndl-PCBs) on neurobehavior of aging people is largely unknown. Therefore, this study aimed toinvestigate the association of ndl-PCBs burden with the cognition functions among elderly adults. Methods Using samples and data from Weitang Geriatric Diseases study (2014–2015), 6 indicator-PCBs were detected in plasma by GC-MS and cognitive dysfunction (CoD) were measured by the Abbreviated Mental Test in 266 participants (age: 61–90). Sequential logistic regression was used to analyze the effects of PCBs on cognition functions. Then the female aged less than and equal to 80 years was selected, and path analysis was used to determine the direct or indirect impacts of co-exposure PCBs on COD by Structural Equation Modeling. Results After sequential adjusting for potential confounder, no association of PCBs 52, 101, 138, 153, 180, LPCBs, HPCBs, and ∑PCBs with the risk of COD was observed; however, the exposure of PCB28 was significantly associated with COD ((Model 3: OR = 3.12, 95% CI = 1.19–8.11, p = 0.020). The final path model also fits that only exposure to PCB28 had a direct effect on COD (β = 0.696, SE = 1.14; p = 0.036). Conclusions After controlling the co-exposures and confounders, the exposure to PCB28 can directly increase the risk of cognitive impairment in older Chinese females.