The abandonment of irrigated paddy land has increased in Japan, which can cause a decline in food security and biodiversity. Despite the importance of individual decisions, most studies have only examined regional or community-based determinants of paddy land abandonment. This study aimed to uncover the socio-economic determinants affecting individual landowners’ decisions to abandon paddy land, using Japanese agricultural census data (2005, 2010, and 2015) composed of over one million unique paddy landowners. Results showed that low agricultural benefits are a key driver of abandonment, similar to European countries. Conversely, there is a positive correlation between the population of full-time cultivators in a household and paddy land abandonment, which contradicts previous evidence. Although some mosaics of socio-ecological landscapes with high biodiversity formed through long-term human influence (i.e., the Satoyama landscapes) are less-favored agricultural areas, the paddy land in some of these landscapes tends not to be abandoned. These findings support effective policymaking that balances biodiversity conservation and the provision of agroecosystem services in semi-natural landscapes.