We study viscosity of the neutron star matter and r-mode instability in rotating neutron stars. Contributions to the shear and bulk viscosities from various processes are calculated accounting for in-medium modifications of the nucleon-nucleon interaction. A softening of the pion mode at densities larger than the nuclear saturation density n 0 and a possibility of the pion condensation at densities above 3n 0 are included. The neutron-neutron and proton-proton pairings are incorporated where necessary. In the shear viscosity we include the lepton contribution calculated taking into account the Landau damping in the photon exchange, the nucleon contribution described by the medium-modified one-pion exchange, and some other terms, such as the novel phonon contribution in the 1S 0 superfluid neutron phase and the neutrino term in the neutrino opacity region. The nucleon shear viscosity depends on the density rather moderately and proves to be much less than the lepton term. On the contrary, among the terms contributing to the bulk viscosity, induced by the delay of the relaxation of lepton concentrations in the star matter perturbed by the r-modes, the term from the two-nucleon medium-modified Urca reactions possesses strongest density dependence (rising by several orders of magnitude for massive stars) because of the pion softening. Also, contributions to the bulk viscosity arising from other reactions induced by charged weak currents, e.g., in the Urca processes on a pion condensate and in direct Urca processes, are included. The radiative bulk viscosity induced by charged and neutral weak currents in the region of the neutrino transparency of the star is also calculated accounting for in-medium effects. We exploit the equation of state, which is similar to the Akmal-Pandharipande-Ravenhall equation of state up to 4n 0 , but is stiffer at higher densities, producing the maximum neutron star mass compatible with observations. The direct Urca processes do not appear up to 5n 0 (corresponding to the star mass M 1.9M ). Computed with account of in-medium effects, the frequency boundary of the r-mode stability for the stars with the mass 1.8M proves to be above the frequencies of all rotating young pulsars. However, none of the conventional contributions to the viscosity are able to explain the stability of rapid rotation of old recycled pulsars in x-ray binaries. To solve this problem we propose a novel efficient mechanism associated with the appearance of condensates of low-lying modes of bosonic excitations with finite momentum and/or with an enhancement of the inhomogeneous pion/kaon condensates in some parts of the star, if the angular velocity exceeds a critical value.