We adopt a modified spin-boson model to investigate the quantum phase transition in an ultracold atom-molecule conversion system involving moleculemolecule interaction. We explore the properties of ground state, entanglement entropy, and many-body dynamics, which confirm that the system exhibits a second-order phase transition from a pure atom phase to a mixed atom-molecule phase when the energy detuning is below a critical value. We obtain three scaling laws and the corresponding two critical exponents to characterize the phase transition. In particular, we discuss the effects of both the speed of ground-state dynamical evolution and the strength of molecular interaction on the phase transition. The adiabatic evolution condition is obtained as well. Our results show that the molecular interaction can greatly reduce the upper bound of the adiabatic condition, which provides a theoretical basis for easier observation of the phase transition in experiments.