The paper analyses possible solutions for mathematical modeling in studies of dynamic processes in dynamically self-locking mechanical systems. The research performed is based on the dynamic system with two degrees of freedom conditioned on the system’s inertial and elastic properties. The orthogonal worm gear with equal helix angles is taken as an example. The paper studies two modes of operation: forward movement and reverse movement. The resulting mathematical model allows determining parameters conditioning self-locking properties of mechanical gears.