The enormous amount of data generated nowadays worldwide is increasingly triggering the search for unconventional and more efficient ways of processing and classifying information, eventually able to transcend the conventional von-Neumann-Turing computational central dogma. It is, therefore, greatly appealing to draw inspiration from less conventional but computationally more powerful systems such as the neural architecture of the human brain.This neuromorphic route has the potential to become one of the most influential and long-lasting paradigms in the field of unconventional computing. The material-based workhorse for current hardware platforms is largely based on standard CMOS technologies, intrinsically following the above mentioned von-Neumann-Turing prescription; we do know, however, that the brain hardware operates in a massively parallel way through a densely interconnected physical network of neurons. This requires challenging the intrinsic definition of the single units and the architecture of computing machines. Memristive and the recently proposed memfractive systems have been shown to display basic features of neural systems such as synaptic-like plasticity and memory features, so that they may offer a diverse playground to implement synaptic connections. Their combination with reservoir computing approaches can further increase their versatility since reservoir networks do not require extra optimization of internal connections. In this review, we address various material-based strategies of implementing unconventional computing hardware: (i) electrochemical oscillators based on liquid metals and (ii) mem-devices exploiting Schottky barrier modulation in polycrystalline and disordered structures made of oxide or perovskite-type semiconductors. Both items (i) and (ii) build the two pillars of neuromimetic computing devices, which we will denote as synthetic neural networks. We complement the more experimental aspects of the review with an overview of few atomistic and phenomenological modelling approaches of memdevices as well as of reservoir computing networks. We expect that the current review will be of great interest for scientists aiming at bridging unconventional computing strategies with specific materials-based platforms.