We study the coupled-top model with three large spins located on a triangle. Depending on the coupling strength, there exist three phases: disordered paramagnetic phase, ferromagnetic phase, and frustrated antiferromagnetic phase, which can be distinguished by the mean-field approach. The paramagnetic-ferromagnetic phase transition is accompanied by the breaking of the global Z2 symmetry, whereas the paramagnetic-antiferromagnetic phase transition is accompanied by the breaking of both the global Z2 symmetry and the translational symmetry. Exact analytical results of higherorder quantum effects beyond the mean-field contribution, such as the excitation energy, quantum fluctuation and von Neumann entropy, can be achieved by the Holstein-Primakoff transformation and symplectic transformation in the thermodynamic limit. Near the quantum critical point, the energy gap closes, along with the divergence of the quantum fluctuation in certain quadrature and von Neumann entropy. Particular attention should be paid to the antiferromagnetic phase, where the geometric frustration takes effect. The critical behaviors in the antiferromagnetic phase are quite different from those in the paramagnetic and ferromagnetic phases, which highlights the importance of the geometric frustration. The triangular coupled-top model provides a simple and feasible platform to study the quantum phase transition and the novel critical behaviors induced by the geometric frustration.