The incorporation of rare earth oxides and nano-silica has been found to significantly enhance the mechanical and tribological characteristics of phenolic-based hybrid nanocomposites. In this work, the impact of these additives was investigated through single-factor experiments. The study revealed that cerium oxide and yttrium oxide were the primary factors influencing changes in the impact strength, shear strength, coefficient of friction, and wear rate. Additionally, the content of nano-silica exerted the most substantial influence on the hardness and compressive strength of the specimens. Furthermore, the material ratios of the phenolic-based hybrid nanocomposites were optimized using an orthogonal experimental design and a fuzzy comprehensive evaluation method. The optimal material ratio for these nanocomposites was determined to be 2% cerium oxide, 2.5% yttrium oxide, and 3% nano-silica, based on their mechanical, frictional, and wear properties. This research provides valuable insights for the development of new brake friction materials with low friction and high wear resistance and contributes to meeting the demand for polymer composites with superior mechanical performance in diverse applications.