Inhibition of mitochondrial axonal trafficking by amyloid beta (Aβ) peptides has been implicated in early pathophysiology of Alzheimer’s Disease (AD). Yet, it remains unclear whether the loss of motility inevitably induces the loss of mitochondrial function, and whether restoration of axonal trafficking represents a valid therapeutic target. Moreover, while some investigations identify Aβ oligomers as the culprit of trafficking inhibition, others propose that fibrils play the detrimental role. We have examined the effect of a panel of Aβ peptides with different mutations found in familial AD on mitochondrial motility in primary cortical mouse neurons. Peptides with higher propensity to aggregate inhibit mitochondrial trafficking to a greater extent with fibrils inducing the strongest inhibition. Binding of Aβ peptides to the plasma membrane was sufficient to induce trafficking inhibition where peptides with reduced plasma membrane binding and internalization had lesser effect on mitochondrial motility. We also found that Aβ peptide with Icelandic mutation A673T affects axonal trafficking of mitochondria but has very low rates of plasma membrane binding and internalization in neurons, which could explain its relatively low toxicity. Inhibition of mitochondrial dynamics caused by Aβ peptides or fibrils did not instantly affect mitochondrial bioenergetic and function. Our results support a mechanism where inhibition of axonal trafficking is initiated at the plasma membrane by soluble low molecular weight Aβ species and is exacerbated by fibrils. Since trafficking inhibition does not coincide with the loss of mitochondrial function, restoration of axonal transport could be beneficial at early stages of AD progression. However, strategies designed to block Aβ aggregation or fibril formation alone without ensuring the efficient clearance of soluble Aβ may not be sufficient to alleviate the trafficking phenotype.