Bat boxes frequently form part of hollow-bearing tree offsets; however, their effectiveness is poorly documented. We investigated the effectiveness of a bat box program designed to partially offset tree hollow loss from clearing for a coal mine. During the first year of monitoring, we detected bats in 5% of 1,308 box checks. Only 3 of 13 local tree cavity-roosting bat species/species groups used boxes and occupancy was not strongly associated with modeled box and site attributes. In the second year, we tested two hypotheses that may explain the relatively low box use: (1) solar exposure of boxes was inadequate for heterothermic bats and (2) available box designs were of low suitability. Relocating boxes to increase solar exposure did not increase use, or enhance the temperature profiles of relocated boxes. Introduction of a new box design led to 11 times higher use compared with existing designs for Nyctophilus spp. (long-eared bat). Overall, our data suggest that the bat box program was ineffective due to few bat species using boxes, infrequent box use by three species, and rarity of maternity roosting. The knowledge gap of species-specific box designs and roosting ecology limits the effectiveness of boxes to offset cleared hollow-bearing trees. Lack of knowledge and the widespread use of bat boxes to offset lost tree hollows highlights the need to (1) rigorously protect hollow-bearing trees and (2) advance our understanding of species-specific roost ecology, box design preferences and mechanical hollow creation into trees, before artificial hollows can be considered a meaningful offset measure.