The properties of nuclear matter and finite nuclei are studied within the quark mean field (QMF) model by taking the effects of pion and gluon into account at the quark level. The nucleon is described as the combination of three constituent quarks confined by a harmonic oscillator potential. To satisfy the spirit of QCD theory, the contributions of pion and gluon on the nucleon structure are treated in second-order perturbation theory. For the nuclear many-body system, nucleons interact with each other by exchanging mesons between quarks. With different constituent quark mass, m q , we determine three parameter sets about the coupling constants between mesons and quarks, named as QMF-NK1, QMF-NK2, and QMF-NK3 by fitting the ground-state properties of several closed-shell nuclei. It is found that all of the three parameter sets can give satisfactory description on properties of nuclear matter and finite nuclei, meanwhile they can also predict the larger neutron star mass around 2.3M ⊙ without the hyperon degrees of freedom.