Olive fruit dreg (OFD), waste from olive softdrink processing, has caused disposal problems. Nevertheless, OFD is a good source of functional ingredients, such as phenolic compounds. This study investigated the extraction conditions of phenolic compounds from OFD by using subcritical water (SCW) extraction method, antioxidant activity of SCW extracts, and components of phenolic compounds by LC-MS. SCW extraction experiments were performed in a batch stainless steel reactor at temperatures ranging from 100 to 180 °C at residence time of 5 to 60 min, and at solid-to-liquid ratio of 1:20 to 1:60. Higher recoveries of phenolic compounds [37.52 ± 0.87 mg gallic acid equivalents (GAE)/g, dry weight (DW)] were obtained at 160 °C, solid-to-liquid ratio of 1:50, and extract time of 30 min than at 2 h extraction with methanol (1.21 ± 0.16 mg GAE/g DW), ethanol (0.24 ± 0.07 mg GAE/g DW), and acetone (0.34 ± 0.01 mg GAE/g DW). The antioxidant activities of the SCW extracts were significantly stronger than those in methanol extracts at the same concentration of total phenolic contents. LC-MS analysis results indicated that SCW extracts contained higher amounts of phenolic compounds, such as chlorogenic acid, homovanillic acid, gallic acid, hydroxytyrosol, quercetin, and syringic acid. SCW at 160 °C, 30 min, and solid-to-liquid ratio of 1:50 may be a good substitute of organic solvents, such as methanol, ethanol, and acetone to recover phenolic compounds from OFD.