Bulk acoustic wave (BAW) resonators are attractive as intermediaries in a microwave-to-optical transducer, due to their long coherence times and controllable coupling to optical photons and superconducting qubits. However, for an optomechanical transducer to operate without detrimental added noise, the mechanical modes must be in the quantum ground state. This has proven challenging in recent demonstrations of transduction based on other types of mechanical resonators, where absorption of laser light caused heating of the phonon modes. In this work, we demonstrate ground state operation of a Brillouin optomechanical system composed of a quartz BAW resonator inside an optical cavity. The system is operated at ∼200 mK temperatures inside a dilution refrigerator, which is made possible by designing the system so that it self-aligns during cooldown and is relatively insensitive to mechanical vibrations. We show optomechanical coupling to several phonon modes and perform sideband asymmetry thermometry to demonstrate a thermal occupation below 0.5 phonons at base temperature. This constitutes the heaviest (∼ 494 µg) mechanical object measured in the quantum ground state to date. Further measurements confirm a negligible effect of laser heating on this phonon occupation. Our results pave the way toward low-noise, high-efficiency microwave-to-optical transduction based on BAW resonators.