Hexagonal boron nitride (BN) is widely used as a substrate and gate insulator for two-dimensional (2D) electronic devices. The studies on insulating properties and electrical reliability of BN itself, however, are quite limited. Here, we report a systematic investigation of the dielectric breakdown characteristics of BN using conductive atomic force microscopy. The electric field strength was found to be ~12 MV/cm, which is comparable to that of conventional SiO2 oxides because of the covalent bonding nature of BN. After the hard dielectric breakdown, the BN fractured like a flower into equilateral triangle fragments. However, when the applied voltage was terminated precisely in the middle of the dielectric breakdown, the formation of a hole that did not penetrate to the bottom metal electrode was clearly observed. Subsequent I-V measurements of the hole indicated that the BN layer remaining in the hole was still electrically inactive. Based on these observations, layer-by-layer breakdown was confirmed for BN with regard to both physical fracture and electrical breakdown. Moreover, statistical analysis of the breakdown voltages using a Weibull plot suggested the anisotropic formation of defects. These results are unique to layered materials and unlike the behavior observed for conventional 3D amorphous oxides.