Brown rice is a familiar whole grain in Japan. We examined national trends in brown rice consumption among Japanese adults aged ≥20 years old, using individual-level data from the National Health and Nutrition Surveys conducted between 2012 and 2019. We employed multivariable logistic regression to identify factors associated with brown rice consumption. The 95th percentile of daily brown rice intake remained at 0.0 g throughout the study period. The percentage of brown rice consumers increased from 1.8% (95% confidence interval: 1.6–2.1) in 2012 to 2.6% (95% confidence interval: 2.0–3.4) in 2019. Compared with individuals who consumed only white rice, brown rice consumers had significantly higher mean intake levels of macronutrients, legumes, vegetables, fruits, and nuts. Brown rice consumption was positively associated with certain sociodemographic characteristics (being female, older age, residing in a major city, living without very young children, and having higher education levels) and health behaviors (lower body mass index, engaging in regular exercise, and being a former or never smoker). Despite its potential nutritional benefits in balanced diets, only a small fraction of adults in Japan consume brown rice, indicating a need for further promotion, particularly among individuals with characteristics associated with brown rice consumption.