The addition of low-loading content of nanofillers may improve the material properties of polymer-based nanocomposites. This improvement directly corresponds to the density of well-dispersed nanofillers in the matrix. However, for higher nanofiller loadings, the nanocomposites' material properties not only may not be improved but also may be degraded due to agglomeration. This complex phenomenon, where nanofillers tend to form agglomerates with the enhancement of volume fraction, poses significant challenges in materials science and nanotechnology. It has been proven that agglomerations hinder the performance of the nanocomposites and thwart the unique properties of nanofillers in most aspects. Graphene, one of the most used nanofillers, plays a remarkable role in nanotechnology. Therefore, the key focus of the current review is to provide insight into the impact of agglomeration on the various material properties such as tensile, flexural, fracture, fatigue, thermal, electrical, and barrier characteristics of the polymer nanocomposites reinforced by graphene-based structures. A comprehensive review of the factors leading to the agglomeration of graphene in the nanocomposites was presented. It was concluded that agglomeration could be a barrier to developing polymer-based nanocomposites, and the challenges of controlling the nanofiller agglomerations were discussed in depth, highlighting the issue's complexity.