Recently, the rights of small-scale fishers have increasingly been acknowledged in ocean governance because coastal development and various maritime activities have reduced traditional fishing grounds. More specifically, small-scale fisheries (SSF) are increasingly being threatened by ocean grabbing, pollution, and a lack of inclusiveness in decision-making processes. Although there are guidelines to resolve and reduce conflict, formal avenues to include fisher concerns, particularly in the context of ocean development and governance, remain a difficult task. Moreover, there is insufficient information on how fishers are impacted by coastal and marine development and how their concerns are included in the decision-making process. Hence, this study contributes to the SSF discourse by understanding and describing the characteristics and concerns of small-scale fishers from two coastal towns in East Africa with different levels of port development. Using data from perception surveys, focus group discussions, and participatory mapping, we discuss how fishers were involved in the decision-making processes to develop ports in Lamu, Kenya, and Bagamoyo, Tanzania. We found that fishers rely on nearshore ecosystems such as mangroves and coral reefs because of their accessibility since most fishers only use low-powered boats for fishing. Moreover, we found that the fishers’ livelihoods were severely affected by port development and that they were excluded from the decision-making process concerning the port’s construction and fishers’ compensations. While some fishers believe that new ports in the region can increase their livelihoods by creating new markets and jobs, this is unlikely to happen since most fishers are not qualified to work in formal port-related jobs. We propose three steps that will allow fishermen to participate in port development decision-making processes and contribute to the development of a sustainable SSF. These include improving engagement with fishers to allow meaningful participation in decision-making, developing a blue economy policy focused on SSF, and implementing maritime spatial planning.