Using computer simulations we investigate the translational and rotational diffusion of dilute suspensions of magnetic nanorods with and without a (homogeneous) external magnetic field. The magnetic rods are represented as spherocylinders with a longitudinal point dipole at their center and length-to-breadth ratiosIn the absence of a field, the rods tend to form compact clusters with antiparallel ordering and thus behave very differently to dipolar spheres (L/D ¼ 0), which tend to form head-to-tail chains. Furthermore, for rod-like particles the external field tends to destabilize rather than to support cluster formation. We show that these differences in the aggregation behavior have profound consequences not only in static material properties such as the field-induced magnetization and the zero-frequency susceptibility, but also in the dynamics. In particular, for magnetic rods the translational diffusion constant parallel to the field is larger than the perpendicular one, in contrast to the behavior observed for magnetic spheres. Moreover, the rod-like character greatly affects the shape and the density dependence of the single-particle and collective dipole-dipole time correlation functions and their counterparts in the frequency domain.