Researches of cellulose nanomaterials have seen nearly exponential growth over the past several decades for versatile applications. The characterization of nanostructural arrangement and local chemical distribution is critical to understand their role when developing cellulose materials. However, with the development of current characterization methods, the simultaneous morphological and chemical characterization of cellulose materials at nanoscale resolution is still challenging. Two fundamentally different nanoscale infrared spectroscopic techniques, namely atomic force microscope based infrared spectroscopy (AFM-IR) and infrared scattering scanning near field optical microscopy (IR s-SNOM), have been established by the integration of AFM with IR spectroscopy to realize nanoscale spatially resolved imaging for both morphological and chemical information. This review aims to summarize and highlight the recent developments in the applications of current state-of-the-art nanoscale IR spectroscopy and imaging to cellulose materials. It briefly outlines the basic principles of AFM-IR and IR s-SNOM, as well as their advantages and limitations to characterize cellulose materials. The uses of AFM-IR and IR s-SNOM for the understanding and development of cellulose materials, including cellulose nanomaterials, cellulose nanocomposites, and plant cell walls, are extensively summarized and discussed. The prospects of future developments in cellulose materials characterization are provided in the final part.