New elastomer-modified brake friction composites with the identical composition, but differing in the type of rubber, were developed by altering polar/non-polar elastomeric phases into the phenolic resin matrix. Effect of dispersion and interaction of the polar/non-polar rubbers with the matrix resin on dry sliding wear characteristics of the friction composites was investigated using a laboratory-scale pin-on-disc tribometer. The composites were prepared by hot mixing followed by compression molding and post-curing at a high temperature. Coefficient of friction (COF) and specific wear rate of the composites sliding against a cast-iron disc were measured and analyzed. Polar acrylonitrile-butadiene rubber (NBR, both powder, and bale rubber), and non-polar styrene-butadiene rubber (SBR) and ethylene-propylene-diene monomer (EPDM) were used with the phenolic resin where rubber islands and its polarity played the key role in delayed stress dissipation, while the hard matrix phase contributed to the intrinsic strength of the composites. This work describes how the distribution and interaction of polar and non-polar rubbers with the matrix resin influence the performance of brake friction composites. NBR (powder)-phenolic composite showed the highest stable average COF of 0.52 as compared to 0.36 for EPDM-based composite. NBR (powder)-based composite also exhibited ~1.5 times better specific wear rate (4.9 × 10 −3 mm 3 N −1 m −1) than EPDM-based composite (7.2 × 10 −3 mm 3 N −1 m −1). However, EPDM rubber-based composite showed a significant enhancement in thermal conductivity (0.43 W/m K). Morphological analyses revealed that the dispersion of rubber phase in phenolic matrix played a significant role in transforming the wear mechanism from abrasive to adhesive.