Face gear transmission is a kind of space-meshing mechanism that is mainly used in the field of aviation. Compared with traditional transmission, it has the advantages of stability, reliability, low noise, and strong carrying capacity. However, owing to its complex tooth surface, there are no means to accurately model the face gear. Likewise, research based on the geometry is difficult. Therefore, the tooth surface equation of the face gear is derived in this article based on the meshing theory. Based on the equations, the point cloud of the face gear tooth surface is calculated, the complex tooth surface is generated, and the face gear is accurately modeled. Moreover, taking tooth surface friction excitation into consideration, a multi-degree-of-freedom nonlinear dynamic model of face gear transmission system is established, using the adaptive variable step length Runge–Kutta method. As shown in the results, the bifurcation diagram, phase diagram, time history diagram, and Poincaré section diagram are combined to analyze the influence of tooth surface friction and meshing frequency on the dynamic characteristics of the system.