Nowadays, the introduction of digital technology improves the condition of the workplace and employees’ productivity, but the unstable behavior of employees is still typical in Internet enterprises in China. Sometimes, employees frequently show their behavior reversals between cooperation and conflict. An integrated analysis method with three steps is performed to explore its reason. First, an evolutionary game model is employed to examine the strategies of individual employee’s behavior selection between cooperation and conflict. Second, the cellular automata are developed to simulate the evolution of employee group behavior selection over time. The frequent behavior turnovers between cooperation and conflict are illustrated. Third, catastrophe theory and method are used to identify the hidden cusp catastrophe patterns under the evolution of group behavior selection. Research results reveal that individual employee selects cooperation if the penalty exceeds half the cost. Simulation results show intense and sudden changes in employee group behavior selection, in which cusp catastrophe patterns exist. The cusp catastrophe model can intuitively interpret the mechanism by which factors, such as average perceived payoff and proportion of cooperation employees, influence the behavior state of the employee group. The mechanism of catastrophe in frequent behavior turnover is explored. This methodology, which is based on the theoretical framework of social exchange theory, integrates evolutionary game theory, simulation, and catastrophe theory to identify the catastrophe mechanism in behavior turnover and make theoretical and practical contributions to behavior selection research.