Olive oil extraction generates several by-products that represent an environmental issue, mainly for Mediterranean countries where olive oil is mostly produced. These by-products represent an ecological issue for their phenolic components, such as oleuropein, hydroxytyrosol, and tyrosol. However, olive oil by-products can be treated and properly exploited in different fields for their health-promoting properties, and they represent great potential for the food and beverage, cosmetic, and pharmaceutical industries. Furthermore, recovery and treatment processes can contribute to efficient waste management, which can enhance the sustainability of the olive oil industry, and in turn, lead to relevant economic benefits. The solid waste, i.e., olive pomace, could be considered to be a suitable matrix or primary resource of molecules with high added value due to their high phenolic content. Olive pomace, at different moisture contents, is the main by-product obtained from two- or three-phase extraction systems. A commonly used centrifugal extraction system, i.e., a multiphase decanter (DMF), does not require the addition of water and can generate a new by-product called pâté or olive pomace cake, consisting of moist pulp that is rich in phenols, in particular, secoiridoids, without any trace of kernel. Although several reviews have been published on olive wastes, only a few reviews have specifically focused on the solid by-products. Therefore, the aim of the present review is to provide a comprehensive overview on the current valorization of the main solid olive oil by-products, in particular, olive pomace or pâté olive cake, highlighting their use in different fields, including human nutrition.