The fabrication of nanocomposites has played role to the development of the nanotechnology and the technology of advanced composite materials. Thermoset polymers are used in engineering applications widely. Their mechanical properties can be change with adding particles. The mechanism of toughening polymers has been suggested recently by reinforcing well dispersed particles to the plain polymer. Nano-silica particles were added to thermoset polymer of polyester to evaluate their influence on the mechanical properties of the toughened polymer using both experimental and numerical methods. The Representative Volume Element (RVE) approach, which employs finite element models, has been developed to achieve that aim numerically for various types of nano-particle reinforcement ratios. In each case, the stiffness has been calculated with using the equivalent homogeneous material concept.  Experimentally, toughened thermoset polymers of polyester reinforced with nano-silica were prepared with different particle content ratio. Several tests were conducted on the nanocomposite, and it was observed increasing nano-silica ratio caused increase in Young’s modulus and decrease in ductility.