This paper presents the effect of the high voltage direct current (HVDC) transmission system based on voltage source converter (VSC) on the sub synchronous resonance (SSR) and low frequency oscillations (LFO) in power system. Also, a novel adaptive neural controller based on neural identifier is proposed for the HVDC which is capable of damping out LFO and sub synchronous oscillations (SSO). For comparison purposes, results of system based damping neural controller are compared with a lead-lag controller based on quantum particle swarm optimization (QPSO). It is shown that implementing adaptive damping controller not only improves the stability of power system but also can overcome drawbacks of conventional compensators with fixed parameters. In order to determine the most effective input of HVDC system to apply supplementary controller signal, analysis based on singular value decomposition is performed. To evaluate the performance of the proposed controller, transient simulations of detailed nonlinear system are considered.