We have observed nonlinear transduction of the thermomechanical motion of a nanomechanical resonator when detected as laser transmission through a sideband unresolved optomechanical cavity. Nonlinear detection mechanisms are of considerable interest as special cases allow for quantum nondemolition measurements of the mechanical resonator's energy. We investigate the origin of the nonlinearity in the optomechanical detection apparatus and derive a theoretical framework for the nonlinear signal transduction, and the optical spring effect, from both nonlinearities in the optical transfer function and second order optomechanical coupling. By measuring the dependence of the linear and nonlinear signal transduction -as well as the mechanical frequency shift -on laser detuning from optical resonance, we provide estimates of the contributions from the linear and quadratic optomechanical couplings.Cavity optomechanics has resulted in new levels of extremely precise displacement transduction [1, 2] of ultrahigh frequency resonators [3]. This has created much interest in pursuing quantum measurements [4] of nanomechancial devices [5][6][7], as well as dynamical back action cooling [8][9][10][11].One of the most fundamental, and as of yet unattained, quantum measurements that could be performed is that of the quantized energy eigenstates of a nanomechanical resonator (as has been demonstrated with an electron in a cyclotron orbit [12]). To achieve this, one cannot measure the displacement of the resonator [13], but instead must measure the energy directly -preferably without destroying the quantum state, a so-called quantum non-demolition (QND) measurement. Whereas the accuracy in continuously measuring two conjugate quantities is limited by the Heisenberg uncertainty principle to the standard quantum limit (SQL) [13], QND measurements allow for continuous measurements of an observable to be taken to arbitrary precision [14][15][16][17]. Here our interest lies in a QND measurement of the energy, and thereby the number of phonons [18]. In an optomechanical system, this is expected to be possible by having strong second order optomechanical coupling [19][20][21][22]. This has been demonstrated in membrane-in-the-middle Fabry-Pérot cavities [23,24], however it has been pointed out there remains first order coupling between the two optical modes, possibly obscuring QND measurements [25].Signal from second order optomechanical coupling, hence measurement of x 2 , will display mechanical peaks at twice the fundamental frequency. However, we would also expect that nonlinear transduction of the displacement, x, of a mechanical resonator from a nonlinear optical transfer function would also appear at harmonics of the mechanical resonance frequency, as has been observed [26][27][28].In this Letter we report observation of peaks in the mechanical power spectra at exactly twice the fundamental mechanical frequency, as shown in Fig. 1. We derive a model for the origin of the harmonic signal, as well as the optical spring effect, from bo...