In this study, a series of new flavones (2-phenyl-chromone), 2-naphthyl chromone, 2-anthryl-chromone, or 2-biphenyl-chromone derivatives containing 6 or 7-substituted tertiary amine side chain were designed, synthesized, and evaluated in acetylcholinesterase (AChE) and butyrylcholinesterase (BChE) inhibition. The results indicated that the alteration of aromatic ring connecting to chromone scaffold brings about a significant impact on biological activity. Compared with flavones, the inhibitory activity of 2-naphthyl chromone, 2-anthryl-chromone derivatives against AChE significantly decreased, while that of 2-biphenyl chromone derivatives with 7-substituted tertiary amine side chain is better than relative flavones derivatives. For all new synthesized compounds, the position of tertiary amine side chain obviously influenced the activity of inhibiting AChE. The results above provide great worthy information for the further development of new AChE inhibitors. Among the newly synthesized compounds, compound 5a is potent in AChE inhibition (IC 50 = 1.29 ± 0.10 μmol/L) with high selectivity for AChE over BChE (selectivity ratio: 27.96). An enzyme kinetic study of compound 5a suggests that it produces a mixed-type inhibitory effect against AChE.