The nonlinearity of a mechanical oscillator may lead to the generation of the macroscopic quantum states, which are useful for precision measurement. Measuring the nonlinearity of a mechanical oscillator becomes important in order to effectively assess its performance. In this paper, we study the electromagnetically induced transparency (EIT) in an optomechanical system with a cubic nonlinear movable mirror. In the presence of the nonlinearity of the movable mirror, we show that the intensity of the output probe field exhibits an asymmetric shape with the transparency peak shifted to a frequency lower than the cavity resonance frequency. This shift can be used to measure the nonlinearity strength of the movable mirror. We also show that the mechanical nonlinearity gives rise to the enhancement of the intensity of the second-order upper sideband generation.

The strong coupling between a macroscopic mechanical oscillator and a cavity field is essential for many quantum phenomena in a cavity optomechanical system. In this work, we discuss the normal mode splitting in a cavity optomechanical system with a cubic nonlinear movable mirror. We study how the mechanical nonlinearity affects the normal-mode splitting behavior of the movable mirror and the output field. We find that the mechanical nonlinearity can increase the peak separation in the spectra of the movable mirror and the output field. We also find that the heights and linewidths of the two peaks are very sensitive to the mechanical nonlinearity.