The utilization of carbonaceous reinforcement-based polymer matrix composites in structural applications has become a hot topic in composite research. Although conventional carbon fiber-reinforced polymer composites (CFRPs) have revolutionized the composite industry by offering unparalleled features, they are often plagued with a weak interface and lack of toughness. However, the promising aspects of carbon fiber-based fiber hybrid composites and hierarchical composites can compensate for these setbacks. This review provides a meticulous landscape and recent progress of polymer matrix-based different carbonaceous (carbon fiber, carbon nanotube, graphene, and nanodiamond) fillers reinforced composites’ mechanical properties. First, the mechanical performance of neat CFRP was exhaustively analyzed, attributing parameters were listed down, and CFRPs’ mechanical performance barriers were clearly outlined. Here, short carbon fiber-reinforced thermoplastic composite was distinguished as a prospective material. Second, the strategic advantages of fiber hybrid composites over conventional CFRP were elucidated. Third, the mechanical performance of hierarchical composites based on carbon nanotube (1D), graphene (2D) and nanodiamond (0D) was expounded and evaluated against neat CFRP. Fourth, the review comprehensively discussed different fabrication methods, categorized them according to performance and suggested potential future directions. From here, the review sorted out three-dimensional printing (3DP) as the most futuristic fabrication method and thoroughly delivered its pros and cons in the context of the aforementioned carbonaceous materials. To conclude, the structural applications, current challenges and future prospects pertinent to these carbonaceous fillers reinforced composite materials were elaborated.