In recent years, agricultural by-product fillers have been investigated in composites to influence the physical properties of the packaging material, increase biodegradability, and reduce costs. In general, the properties of composites are mainly influenced by the type, amount, and size of fillers. The aim of this study was to characterize potato peel particles as a filler in a poly(lactic acid) (PLA) matrix and to determine the effect of particle size on the physical properties of the composite. Therefore, different fractions of potato peel powder (0–53 μm, 125–250 μm, and 315–500 μm) were incorporated into PLA matrix via compounding and injection-molding. Microscopic analysis of the injection-molded samples revealed that the average particle shape did not differ between the different fractions. Overall, increasing the particle size of potato peel particles resulted in increased stiffness and decreased ductility. The cold crystallization temperature and water vapor transmission rate of the composites were independent of particle size but increased upon the incorporation of potato peel particles. In conclusion, the effect of particle incorporation on packaging-related properties was higher than the effect of using different particle size fractions. This means that potato peel particles, regardless of their particle size distribution, are promising fillers for composites, with the potential to improve biodegradability, maintain some level of protection for the packaged product, and reduce the cost of the composites.