The use of plant aggregates obtained from agricultural co-products mixed with mineral binders to form eco-friendly insulating building materials has been initiated for a few years to bring environmentally friendly solutions to the construction sector. Several studies on different agro-resources have already been carried out, providing various information about the properties of plant aggregates and plant-based concrete. However, the characteristics of the agricultural co-product, which allow it to qualify as a plant aggregate for plant-based concrete, are not yet very clear despite the multitude of data, especially on hemp concrete. Therefore, it is important to gather numerous but very disparate pieces of information available in the literature concerning the properties of plant aggregates and their correlations with composites. This review is based on the results of 120 articles and aims to identify the characterization methods and the multi-physical properties of plant aggregates affecting those of plant-based concrete and to propose additional factors that could influence the properties of the composites. A total of 18 plant aggregates of different origins used for plant-based concrete have been listed in the literature. In France, hemp shiv is the most studied one, but its quantity is quite low unlike cereal or oilseed straws and wood transformation residues. With the existence of several characterization methods, properties like microstructure, particle size distribution, bulk density, water absorption capacity, and chemical composition of aggregates are easily and frequently determined. In contrast, data on the apparent density of particles, the skeleton density, and the hygro-thermal properties of aggregates are rare. The particle size, density, and porosity have been identified as important parameters influencing the properties of the composites. Other parameters related to the behavior of the aggregates under wet compaction and compression of their stacking can also predict the physical and mechanical properties of the obtained plant-based concrete. Dosages of the constituents should be preferred as formulation parameters for future studies assessing the impact of the aggregate properties on the composites.