At nanometer separation, the dominant interaction between an atom and a material surface is the fluctuation-induced Casimir-Polder potential. We demonstrate that slow atoms crossing a silicon nitride transmission nanograting are a remarkably sensitive probe for that potential. A 15% difference between nonretarded (van der Waals) and retarded Casimir-Polder potentials is discernible at distances smaller than 51 nm. We discuss the relative influence of various theoretical and experimental parameters on the potential in detail. Our work paves the way to high-precision measurement of the Casimir-Polder potential as a prerequisite for understanding fundamental physics and its relevance to applications in quantum-enhanced sensing.