In recent years, the use of waste materials from agricultural sources has attracted interest as a research field. Several kinds of waste particles have been studied as additives for asphalt modification, and good results in terms of rheological and physical properties have been discussed in recent literature. In the present work, two types of seed shell particles were evaluated as asphalt modifiers. The shells of Jatropha curcas and pistachio are considered waste materials with no further industrial applications; therefore, in this study, they were incorporated into asphalt at different concentrations in order to evaluate their effect on the binder’s properties. Modified asphalt mixtures were prepared through the hot mix method, and the physical, rheological, and thermal properties of the modified samples were measured and compared to those of the unmodified binder. According to the results of softening point, viscosity, and rheological characterization, the particles obtained from Jatropha curcas and pistachio shells can be used as efficient additives for asphalt modification. Pistachio shell particles act as an asphalt modifier, while Jatropha curcas behaves more like a filler agent since using it at high concentrations causes an inverse effect on the modified performance and properties of the asphalt. Finally, the results obtained showed that both shell particles were useful for improving the binder’s resistance to rutting and permanent deformations, compared to the pure asphalt’s original behavior.