Heat stress occurring during rice (Oryza sativa) grain development reduces grain quality, which often manifests as increased grain chalkiness. Although the impact of heat stress on grain yield is well-studied, the genetic basis of rice grain quality under heat stress is less explored as quantifying grain quality is less tractable than grain yield. To address this, we used an image-based colorimetric assay (Red, R; and Green, G) for genome-wide association analysis to identify genetic loci underlying the phenotypic variation in rice grains exposed to heat stress. We found the R to G pixel ratio (RG) derived from mature grain images to be effective in distinguishing chalky grains from translucent grains derived from control (28/24°C) and heat stressed (36/32°C) plants. Our analysis yielded a novel gene, rice Chalky Grain 5 (OsCG5) that regulates natural variation for grain chalkiness under heat stress. OsCG5 encodes a grain-specific, expressed protein of unknown function. Accessions with lower transcript abundance of OsCG5 exhibit higher chalkiness, which correlates with higher RG values under stress. These findings are supported by increased chalkiness of OsCG5 knock-out (KO) mutants relative to wildtype (WT) under heat stress. Grains from plants overexpressing OsCG5 are less chalky than KOs but comparable to WT under heat stress. Compared to WT and OE, KO mutants exhibit greater heat sensitivity for grain size and weight relative to controls. Collectively, these results show that the natural variation at OsCG5 may contribute towards rice grain quality under heat stress.