Once a beach is eroded by storm waves, it is generally recovered under milder wave conditions. To prevent or reduce damage, it is therefore important to understand the characteristics of the site-specific recovery process. Here, we present the results, based on a data set from a video monitoring system and wave measurements, of the recovery process in a pocketed beach located inside a bay where the shoreline retreated harshly (~12 m, on average, of beach width) during Typhoon TAPAH (T1917) in September 2019. It took about 1.5 years for the beach to be recovered to the level before the typhoon. During this period, the erosion and accretion were repeated, with the pattern highly related to the wave power (Pw); most of the erosion occurred when Pw became greater than 30 kWatt/m, whereas the accretion prevailed when Pw was no greater than 10 kWatt/m. The recovery pattern showed discrepancies between different parts of the beach. The erosion during storm events was most severe in the southern part, whereas the northern shoreline did not significantly change even during TAPAH (T1917). In contrast, the recovery process occurred almost equally at all locations. This discrepancy in the erosion/accretion process was likely due to human intervention, as a shadow zone was formed in the northern end due to the breakwaters, causing disequilibrium in the sediment transport gradient along the shore. The results in this study could be applied in designing the protection plans from severe wave attacks by effectively estimating the size of coastal structures and by correctly arranging the horizontal placement of such interventions or beach nourishment. Although the application of these results should be confined to this specific site, the method using wave energy parameters as criteria can be considered in other areas with similar environments, for future planning of beach protection.