Eco-friendly composites are proposed to substitute commonly available polymers. Currently, wood–plastic composites and natural fiber-reinforced composites are gaining growing recognition in the industry, being mostly on the thermoplastic matrix. However, little data are available about the possibility of producing biocomposites on a silicone matrix. This study focused on assessing selected organic fillers’ impact (ground coffee waste (GCW), walnut shell (WS), brewers’ spent grains (BSG), pistachio shell (PS), and chestnut (CH)) on the physicochemical and mechanical properties of silicone-based materials. Density, hardness, rebound resilience, and static tensile strength of the obtained composites were tested, as well as the effect of accelerated aging under artificial seawater conditions. The results revealed changes in the material’s properties (minimal density changes, hardness variation, overall decreasing resilience, and decreased tensile strength properties). The aging test revealed certain bioactivities of the obtained composites. The degree of material degradation was assessed on the basis of the strength characteristics and visual observation. The investigation carried out indicated the impact of the filler’s type, chemical composition, and grain size on the obtained materials’ properties and shed light on the possibility of acquiring ecological silicone-based materials.