Human decision-making does not conform to the independent decision-making hypothesis from classical decision-making theory. Thus, we introduce quantum decision-making theory into the Lotka–Volterra model (L–V model), to investigate player population dynamics while incorporating the initial strategy, game payoffs and interactive strategies in an open social system. Simulation results show that: (1) initial strategy, entanglement intensity of strategy interaction, and payoffs impact population dynamics; (2) In cooperative coexistence, game players mutually exceed the initial environmental capacity in an open system, but not in competitive coexistence; (3) In competitive coexistence, an initial strategy containing an entanglement intensity of strategies plays a vital role in game outcomes. Furthermore, our proposed model more realistically delineates the characteristics of population dynamics in competitive or cooperative coexistence scenarios.