Neuromorphic computing is assumed to be significantly more energy efficient than, and at the same time expected to outperform, conventional computers in several applications, such as data classification, since it overcomes the so-called von Neumann bottleneck. Artificial synapses and neurons can be implemented into conventional hardware using new software, but also be created by diverse spintronic devices and other elements to completely avoid the disadvantages of recent hardware architecture. Here, we report on diverse approaches to implement neuromorphic functionalities in novel hardware using magnetic elements, published during the last years. Magnetic elements play an important role in neuromorphic computing. While other approaches, such as optical and conductive elements, are also under investigation in many groups, magnetic nanostructures and generally magnetic materials offer large advantages, especially in terms of data storage, but they can also unambiguously be used for data transport, e.g., by propagation of skyrmions or domain walls. This review underlines the possible applications of magnetic materials and nanostructures in neuromorphic systems.