A series of composites based on epoxy resin filled with additives of natural origin were prepared to investigate the influence of such fillers on the properties of the epoxy compositions. For this purpose, the composites containing 5 and 10 wt.% of additive of natural origin were obtained using the dispersion of oak wood waste and peanut shells in bisphenol A epoxy resin cured with isophorone-diamine. The oak waste filler had been obtained during the assembly of the raw wooden floor. The performed studies include testing of samples prepared using unmodified and chemically modified additives. Chemical modification via mercerization and silanization was performed to increase the poor compatibility between the highly hydrophilic fillers of natural origin and the hydrophobic polymer matrix. Additionally, the introduction of NH2 groups to the structure of modified filler via 3-aminopropyltriethoxysilane, potentially takes a part in co-crosslinking with the epoxy resin. Fourier Transformed Infrared Spectroscopy (FT–IR), as well as Scanning Electron Microscopy (SEM), were carried out, to study the influence of performed chemical modification on the chemical structure and morphology of wood and peanut shell flour. SEM analyses showed significant changes in the morphology of compositions with chemically modified fillers, indicating improved adhesion of the resin to lignocellulosic waste particles. Moreover, a series of mechanical (hardness, tensile strength, flexural strength, compressive strength, and impact strength) tests were carried out, to assess the influence of the application of fillers of natural origin on the properties of epoxy compositions. All composites with lignocellulosic filler were characterized by higher compressive strength (64.2 MPa—5%U-OF, 66.4%—SilOF, 63.2—5%U-PSF, and 63.8—5%SilPSF, respectively), compared to the values recorded for the reference epoxy composition without lignocellulosic filler (59.0 MPa—REF). The highest compressive strength, among all tested samples, was recorded for the composite filled with 10 wt.% of unmodified oak flour (69.1 MPa—10%U-OF). Additionally, higher values of flexural and impact strength, concerning pure BPA-based epoxy resin, were recorded for the composites with oak filler (respectively, flexural strength: 73.8 MPa—5%U-OF and 71.5 MPa—REF; impact strength: 15.82 kJ/m2—5%U-OF, 9.15 kJ/m2—REF). Epoxy composites with such mechanical properties might be considered as broadly understood construction materials. Moreover, samples containing wood flour as a filler exhibit better mechanical properties compared to those with peanut shell flour (tensile strength for samples containing post-mercerization filler: 48.04 MPa and 40.54 MPa; while post-silanization 53.53 MPa and 42.74 MPa for compositions containing 5 wt.% of wood and peanut shell flour, respectively). At the same time, it was found that increasing the weight share of flour of natural origin in both cases resulted in the deterioration of mechanical properties.