This study aimed to investigate the behavior, and failure resistance of composite materials, and the effect of nanoparticles on their flexural properties. Nanoparticles with different weight percentages of nano-clay, and nano-silica were added to a plain weave C-glass fabric/epoxy/nano SiO2/nano-clay hybrid composite to analyze their effect. A three-point bending test was used to determine the findings. Initially, nano-clay and nano-silica were dispersed into epoxy resin at loadings of 0, 1, 3, and 5 weight percent for all hybrid composites using ultrasonic and mechanical stirring techniques. The test results show that the addition of both these nanoparticles up to 1 wt.% improved flexural strength, and flexural modulus by 12% and 26%, for nano-clay, 13% and 29%, for nano-silica, respectively. Then, during three-point bending tests, acoustic emission (AE) technique was used to detect damage mechanisms. Afterward, to classify damage mechanisms using AE features (peak amplitude and frequency), Gaussian Mixture Model (GMM) was used. Additionally, the evolution of damages in samples containing and without nanoparticles was investigated and studied using cumulative AE energy. The results show that the samples containing nanoparticles were more resistant to the growth of monotonous damage than those without nanoparticles.