We describe the construction of an apparatus designed to realize a hybrid quantum system comprised of a cryogenically cooled mechanical oscillator and ultra-cold Rb atoms coupled via light. The outstanding feature of our instrument is an in situ adjustable asymmetric all-fiber membrane-in-the-middle cavity located inside an ultra-high vacuum dilution refrigerator based cryostat. We show that Bose-Einstein condensates of N=2×10 atoms can be produced in less than 20 s and demonstrate a single photon optomechanical coupling strength of g=2π×9 kHz employing a high-stress SiN membrane with a mechanical quality factor Q>10 at a cavity setup temperature of T = 480 mK.

A promising route to novel quantum technologies are hybrid quantum systems, which combine the advantages of several individual quantum systems. We have realized a hybrid atomic-mechanical experiment consisting of a Si 3 N 4 membrane oscillator cryogenically precooled to 500 mK and optically coupled to a cloud of laser cooled 87 Rb atoms. Here, we demonstrate active feedback cooling of the oscillator to a minimum mode occupation of n 16 1 m =  corresponding to a mode temperature of T min ≈200 μK. Furthermore, we characterize in detail the coupling of the membrane to the atoms by means of sympathetic cooling. By simultaneously applying both cooling methods we demonstrate the possibility of preparing the oscillator near the motional ground state while it is coupled to the atoms. Realistic modifications of our setup will enable the creation of a ground state hybrid quantum system, which opens the door for coherent quantum state transfer, teleportation and entanglement as well as quantum enhanced sensing applications.

We study and realize asymmetric fiber-based cavities with optimized mode match to achieve high reflectivity on resonance. This is especially important for mutually coupling two physical systems via light fields, e.g. in quantum hybrid systems. Our detailed theoretical and experimental analysis reveals that on resonance the interference effect between the directly reflected non-modematched light and the light leaking back out of the cavity can lead to large unexpected losses due to the mode filtering of the incoupling fiber. Strong restrictions for the cavity design result out of this effect and we show that planar-concave cavities are clearly best suited. We validate our analytical model using numerical calculations and demonstrate an experimental realization of an asymmetric fiber Fabry-Pérot cavity with optimized parameters.