Advances in cellular recordings using engineered devices have greatly increased our understanding in biological systems. These recordings have been an essential part in a broad variety of fields ranging from neuroscience and cardiology to cell biology and have been applied both in research and clinical applications. To provide less tissue damage, reduce foreign-body response, and enhance long-term signal stability, the engineered devices need to be minimally invasive. Pursuing these goals, recently, nanoelectronic probes with improved mechanical, chemical, and biocompatible properties have been developed. Moreover, innovative and automated insertion techniques of neural probes have been reported, further progressing towards a high-bandwidth brain-machine interface system. In this progress report, recent progress in materials, device designs, and implementation techniques aiming for minimally invasive cellular recordings (both in vivo and in vitro) is highlighted, and future directions in these areas are proposed.