Self-oscillations are one of the common problems in the complex automatic system, that can occur due to the features of the workflow and the design of the governor. The development of digital control systems has made it possible to damp self-oscillations by applying complex control laws. However, for hydromechanical systems, such way is unacceptable due to the design complexity and the governor cost. The objective of this work is to determine the parameters of the hydromechanical free turbine speed controller, ensuring the absence of self-oscillations during ground tests of the turboshaft engine with a hydraulic dynamometer. The TV3-117VM engine (Ukraine) with the NR-3VM regulator pump (Ukraine) was selected as the object of the study. However, self-oscillations can also occur in any modifications of the TV3-117 engine with any NR-3 regulator pump. The results of the research may be of interest to engineers and scientists who investigate the dynamics of automatic control systems for similar engines. The paper analyses the nonlinear features of the empirical characteristics of the FTSC leading to self-oscillations of the engine speed. The authors propose the mathematical model of the automatic control system dynamics, which takes into account all the features of the engine and regulator pump. It is shown that the load characteristics of the water brake and the helicopter main rotor can differ significantly. Research of the dynamic characteristics of the TV3-117VM engine was carried out. The analysis showed a good agreement between the calculation results and the field test results, and made it possible to determine the parameters of the controller, which lead to self-oscillations during test. Two cases are considered. The first case includes ground tests of the engine with a water brake; the second case—flight tests of the engine as part of the helicopter’s power plant. The data obtained make it possible to develop recommendations for adjusting the hydromechanical governor without testing it on the engine.