A nonlinear dynamic model of Bending-Torsional-Axial-Pendular (BTAP) has been developed for a Coaxial Reverse Closed Differential Herringbone Gear Transmission System (CRCDHGTS) with consideration of gear floating. This model takes into account factors such as gear floating backlash, tooth surface friction, gyroscopic effect, Time Varying Meshing Stiffness (TVMS), meshing damping, and dynamic meshing parameters. To investigate the impact of gear floating on the nonlinear dynamic characteristics of the system, a gear floating model was developed using the concept of gear floating. The calculations included determining gear floating backlash and TVMS with consideration for gear floating. The impact of input speed, initial backlash, gear float value, and system transmission error on the nonlinear dynamic vibration characteristics is analyzed using various diagrams including bifurcation diagram, Maximum Lyapunov Exponent (MLE), time history diagram, frequency diagram, phase diagram, and Poincaré section diagram. The research reveals that gear floating diminish the chaotic motion behavior of the system under different excitation factors, improving the system's global bifurcation characteristics. The developed BTAP coupled nonlinear dynamic model provides more accurate numerical solutions compared to models with fixed meshing parameters, making it more suitable for analyzing the system's dynamic characteristics. Analysis of the gear floating value indicates an optimal range of 0-20μm and 34-43μm for generating periodic motion, with floating values around 10-20μm showing better performance in reducing the negative effects of initial backlash and transmission error.