We developed a refined gravimetric geoid model for Japan on a 1 × 1.5 arc-minute (2 km) grid from a GOCE-based satellite-only global geopotential model and a regional gravity field model updated in this study. First, we have constructed a regional gravity field model for Japan using updated gravity datasets together with a residual terrain model: 323,431 land gravity data, 77,389 shipborne marine gravity data, and Sandwell's v28.1 altimetry-derived global marine gravity model. Then, the geoid was determined with the gravity field model. The methodology for gravimetric geoid determination was based on the remove-compute-restore technique with Helmert's second method of condensation of topography (Stokes-Helmert scheme). Here, the hybrid Meissl-Molodensky modified spheroidal Stokes kernel was employed to minimize the truncation error under an appropriate combination of different kinds of gravity data. In addition, a high-resolution GSI-DEM on a 0.4 × 0.4 arc-second (10 m) grid, together with the SRTM-DEM on a 7.5 × 11.25 arc-second (250 m) grid, was utilized for precisely applying terrain correction to the regional gravity field model. Consequently, we created a gravimetric geoid model for Japan, consistent with 971 GNSS/leveling geoid heights distributed over the four main islands of Japan with a standard deviation of 5.7 cm, showing a considerable improvement by 2.3 cm over the previous model (JGEOID2008). However, there remain some areas with large discrepancies between the computed and GNSS/leveling geoid heights in northern Japan (Hokkaido), mountainous areas in central Japan, and some coastal regions. Since terrestrial gravity data are especially sparse in these areas, we speculated that the largeness of the geoid discrepancies there could be partly attributed to the insufficient coverage and accuracy of gravity data. The Geospatial Information Authority of Japan has started airborne gravity surveys to be covered over the Japanese Islands, and in future, we plan to develop a geoid model for Japan further accurately by incorporating airborne gravity data to come.