Biological nitrogen fixation is a major factor contributing to microbial primary productivity in the open ocean. The current view depicts a few cyanobacterial diazotrophs as the most relevant marine nitrogen fixers, whereas heterotrophic diazotrophs are more diverse and considered to have lower impacts on the nitrogen balance. Here, we used 891 Tara Oceans metagenomes to create a manually curated, non-redundant genomic database corresponding to free-living, as well as filamentous, colony-forming, particle-attached and symbiotic bacterial and archaeal populations occurring in the surface of five oceans and two seas. Notably, the database provided the genomic content of eight cyanobacterial diazotrophs including Trichodesmium populations and a newly discovered population similar to Richelia, as well as 40 heterotrophic bacterial diazotrophs organized into three main functional groups that considerably expand the known diversity of abundant marine nitrogen fixers compared to previous genomic surveys. Critically, these 48 populations may account for more than 90% of cells containing known nifH genes and occurring in the sunlit ocean, suggesting that the genomic characterization of the most abundant marine diazotrophs may be nearing completion. The newly identified heterotrophic bacterial diazotrophs are widespread, express their nifH genes in situ, and co-occur under nitrate-depleted conditions in large size fractions where they might form aggregates providing the low-oxygen microenvironments required for nitrogen fixation. Most significantly, we found heterotrophic bacterial diazotrophs to be more abundant than cyanobacterial diazotrophs in most metagenomes from the open oceans and seas. This large-scale environmental genomic survey emphasizes the considerable potential of heterotrophs in the marine nitrogen balance.