A quantum theory of cooling of a mechanical oscillator by radiation pressure-induced dynamical back-action is developed, which is analogous to sideband cooling of trapped ions. We find that final occupancies well below unity can be attained when the mechanical oscillation frequency is larger than the cavity linewidth. It is shown that the final average occupancy can be retrieved directly from the optical output spectrum.PACS numbers: 42.65. Sf ,42.65.Ky, 42.79.Gn Mesoscopic mechanical oscillators are currently attracting interest due to their potential to enhance the sensitivity of displacement measurements [1] and to probe the quantum to classical transition of a macroscopic degree of freedom [2,3]. A prerequisite for these applications is the capability of initializing an oscillator with a long phonon lifetime in its quantum ground state. So far this has not been demonstrated because the combination of sufficiently high mechanical frequencies (ω m /2π) and quality factors in the relevant regime ω m ≫ k B T has not been reached [3]. In contrast, in atomic physics laser cooling has enabled the preparation of motional ground states [4,5]. This has prompted researchers to study means of cooling a single mechanical resonator mode directly using laser radiation. Early work demonstrated cooling of a mechanical degree of freedom of a Fabry-Pérot mirror using a radiation pressure force controlled by an electronic feedback scheme [6,7], in analogy to stochastic cooling. In contrast, the radiation pressure induced coupling of an optical cavity mode to a mechanical oscillator [cf. Fig. 1(a)] can give rise to self-cooling via dynamical back-action [8]. In essence, the cavity delay induces correlations between the radiation pressure force and the thermal Brownian motion that lead to cooling or amplification, depending on the laser detuning. In a series of recent experiments, these effects have been used to cool a single mechanical mode [9,10,11]. While classical and semiclassical analysis of dynamical back-action have been developed [13,14], the question as to whether ground state cooling is possible has not been addressed.Here a quantum theory of cooling via dynamical backaction is presented. We find that final occupancies below unity can indeed be attained when the optical cavity's lifetime is comparable to or exceeds the mechanical oscillation period. Along these lines, an analogy between this mechanism and the sideband cooling of trapped ions in the Lamb-Dicke regime is elucidated [5]. In our setting the optical cavity mode plays the role of the ion's pseudospin mediating the frequency up-conversion underlying the cooling cycle. Finally, we discuss how the average phonon occupancy can be retrieved from the spectrum of the optical cavity output. We note that these results can be applied to a wide range of experimental realizations of cavity self-cooling [9,11,12].We treat the laser driven optical cavity mode coupled to the mechanical resonator mode as an open quantum system and adopt a rotating frame at the laser freq...