Mangroves are adapted to coastal processes; however, mangrove species showed various responses to estuarian environments, leading to different structural characteristics at accretion and erosion areas. The species composition, structure and regeneration of mangrove forests were investigated to provide insight into mangrove forest development in response to shoreline accretion and erosion processes. The species composition and stand structure of mangrove forests were measured along the distance from the shoreline at accretion and erosion sites in Ca Mau, Vietnam. The hierarchical clustering of grouped stands based on species composition and tree size distribution was conducted. Grouped mangrove stands showed landward changes in species composition and stand structure from the shoreline (p < 0.05), reflecting the timescale of accretion or erosion at both accretion and erosion sites. Stand development patterns differed between accretion and erosion sites, and Avicennia alba and Rhizophora apiculata dominated seaward plots at accretion and erosion sites, respectively. Newer accredited sites were dominated by A. alba. Mangrove stands developed from dense A. alba dominant to R. apiculata dominant stands with increasing tree size at accretion sites. There were more species-colonized sites with a higher erosion rate or that were more recently eroded, implying that timescale of erosion and erosion rate affected species composition and regeneration on erosion sites. Accretion and erosion affected stand development of mangroves differently, implying that conservation and restoration strategies should be applied differently to accretion and erosion sites.