Cereal grains and nuts are represented as the economic backbone of many developed and developing countries. Kernels of cereal grains and nuts are prone to mold infection under high relative humidity and suitable temperature conditions in the field as well as storage conditions. Health risks caused by molds and their molecular metabolite mycotoxins are, therefore, important topics to investigate. Strict regulations have been developed by international trade regulatory bodies for the detection of mold growth and mycotoxin contamination across the food chain starting from the harvest to storage and consumption. Molds and aflatoxins are not evenly distributed over the bulk of grains, thus appropriate sampling for detection and quantification is crucial. Existing reference methods for mold and mycotoxin detection are destructive in nature as well as involve skilled labor and hazardous chemicals. Also, these methods cannot be used for inline sorting of the infected kernels. Thus, analytical methods have been extensively researched to develop the one that is more practical to be used in commercial detection and sorting processes. Among various analytical techniques, optical imaging and spectroscopic techniques are attracting growers' attention for their potential of nondestructive and rapid inline identification and quantification of molds and mycotoxins in various food products. This review summarizes the recent application of rapid and nondestructive optical imaging and spectroscopic techniques, including digital color imaging, X-ray imaging, near-infrared spectroscopy, fluorescent, multispectral, and hyperspectral imaging. Advance chemometric techniques to identify very low-level mold growth and mycotoxin contamination are also discussed. Benefits, limitations, and challenges of deploying these techniques in practice are also presented in this paper.