Thermoelectric devices (TEDs), including thermoelectric generators (TEGs) and thermoelectric coolers (TECs) based on the Seebeck and Peltier effect, respectively, are capable of converting heat directly into electricity and vice versa. Tough suffering from low energy conversion efficiency and relatively high capital cost, TEDs have found niche applications, such as the remote power source for spacecraft, solid-state refrigerators, waste heat recycling, and so on. In particular, on-chip integrable micro thermoelectric devices (μ-TEDs), which can realize local thermal management, on-site temperature sensing, and energy harvesting under minor temperature gradient, could play an important role in biological sensing and cell cultivation, self-powered Internet of Things (IoT), and wearable electronics. In this review, starting from the basic principles of thermoelectric devices, we summarize the most critical parameters for μ-TEDs, design guidelines, and most recent advances in the fabrication process. In addition, some innovative applications of μ-TEDs, such as in combination with microfluidics and photonics, are demonstrated in detail.