This paper investigates the effect of sub‐micron size cenosphere filler and filler loading on mechanical and dry sliding wear property of polyester composites. Composites are fabricated by filling with 10 and 20 wt% of 800 and 200‐nm size of cenosphere filler particles. Neat polyester composite is also prepared for comparison analysis. Dry sliding wear test is conducted for these composites over a range of sliding distance with different sliding velocities and applied loads on a pin‐on‐disc wear test machine. Taguchi methodology and analysis of variance (ANOVA) is used to analyze the friction and wear characteristics of the composites. The artificial neural network (ANN) approach is implemented to the friction and wear data for corroboration. In this work, mechanical properties of composites such as hardness, tensile strength, tensile modulus, flexural strength, and compressive strength revealed that mechanical properties and wear resistance of the composites increase with a decrease in the particle size. The measured Young's moduli are comparable to standard theoretical prediction models. The morphology of worn composite specimens has been examined by scanning electron microscopy to understand the dominant wear mechanisms. Finally, optimal factor settings are determined using a genetic algorithm (GA). Copyright © 2017 John Wiley & Sons, Ltd.