In downhole drilling systems, self-excited torsional vibrations caused by the bit-rock interactions can affect the drilling process and lead to the premature failure of components. Especially self-excited oscillations of higher-order modes lead to critical dynamic loads. The slim drill string design and the naturally limited drilled borehole diameter limit the installation space, power supply and lead to numerous potentially critical self-excited torsional modes. Consequently, small and robust passive damping concepts are required. The variety of possible downhole boundary conditions and potential damper designs necessitates analytical solutions for effective damper design and optimization. In this paper, two nonlinear passive damper concepts are investigated regarding design and effectiveness to reduce self-excited high-frequency torsional oscillations in drill string dynamics. Based on a finite element model of a drill string, a suitable minimal model based on the identified critical mode is generated and solved analytically using the Multiple Scales Lindstedt-Poincaré (MSLP) method. The advantages of MSLP compared to conventional MS methods are shown for this example. On the basis of the analytical solution, parameter influences are determined, and design equations are derived. The analytical results are transferred to self-excited drill string vibrations and discussed using time domain simulations of the drill string model.