Fishery improvement projects (FIPs) are emerging as a popular market‐based means to improve fisheries sustainability and have been employed in scores of fisheries around the world; however, project ability to realize improvements has been highly variable, and little is known about how fishery and project conditions affect improvement efforts. In order to evaluate the effectiveness of the FIP model as a tool for improving diverse fisheries around the world, we compile a unique dataset of social, ecological and economic characteristics for over 60 FIPs globally, which we use to identify key attributes correlated with improvements in fishing practices, management and/or on‐the‐water outcomes. Using a random forest classifier, we identify three important attributes related to FIP effectiveness in demonstrating improvements. Specifically, FIPs are more likely to have achieved improvements with increased cumulative project time, when regional‐level management arrangements are present and when the target species has a moderate inherent vulnerability to fishing. Interestingly, improvements were not correlated with a number of expected features, including a fishery's socio‐economic setting or baseline performance against the desired sustainability standard (e.g. the Marine Stewardship Council fisheries standard). This study improves our understanding of factors related to FIP effectiveness in improving fisheries practices and management and provides key insights for practitioners into important attributes to consider when implementing the FIP model to promote fisheries sustainability.