Polymer-bonded magnets are a class of composite material that combines the magnetic properties of metal particles and the molding possibility of a polymeric matrix. This class of materials has shown huge potential for various applications in industry and engineering. Traditional research in this field has so far mainly focused on mechanical, electrical or magnetic properties of the composite, or on particle size and distribution. This examination of synthesized Nd-Fe-B-epoxy composite materials includes the mutual comparison of impact toughness, fatigue, and the structural, thermal, dynamic-mechanical, and magnetic behavior of materials with different content of magnetic Nd-Fe-B particles, in a wide range from 5 to 95 wt.%. This paper tests the influence of the Nd-Fe-B content on impacting the toughness of the composite material, as this relationship has not been tested before. The results show that impact toughness decreases, while magnetic properties increase, along with increasing content of Nd-Fe-B. Based on the observed trends, selected samples have been analyzed in terms of crack growth rate behavior. Analysis of the fracture surface morphology reveals the formation of a stable and homogeneous composite material. The synthesis route, the applied methods of characterization and analysis, and the comparison of the obtained results can provide a composite material with optimum properties for a specific purpose.