In recent years, food safety incidents have been frequently reported. Food or raw materials themselves contain substances that may endanger human health and are called toxic and harmful substances in food, which can be divided into endogenous, exogenous toxic, and harmful substances and biological toxins. Therefore, realizing the rapid, efficient, and nondestructive testing of toxic and harmful substances in food is of great significance to ensure food safety and improve the ability of food safety supervision. Among the nondestructive detection methods, infrared spectroscopy technology has become a powerful solution for detecting toxic and harmful substances in food with its high efficiency, speed, easy operation, and low costs, while requiring less sample size and is nondestructive, and has been widely used in many fields. In this review, the concept and principle of IR spectroscopy in food are briefly introduced, including NIR and FTIR. Then, the main progress and contribution of IR spectroscopy are summarized, including the model’s establishment, technical application, and spectral optimization in grain, fruits, vegetables, and beverages. Moreover, the limitations and development prospects of detection are discussed. It is anticipated that infrared spectroscopy technology, in combination with other advanced technologies, will be widely used in the whole food safety field.