Interspecific hybridization is widespread in nature and can result in the formation of new hybrid species as well as the transfer of traits between species. However, the fate of newly formed hybrid lineages is relatively understudied. We undertook pairwise crossing between multiple genotypes of three Brassica allotetraploid species Brassica juncea (2n = AABB), Brassica carinata (2n = BBCC), and Brassica napus (2n = AACC) to generate AABC, BBAC, and CCAB interspecific hybrids and investigated chromosome inheritance and fertility in these hybrids and their self-pollinated progeny. Surprisingly, despite the presence of a complete diploid genome in all hybrids, hybrid fertility was very low. AABC and BBAC first generation (F1) hybrids both averaged ~16% pollen viability compared to 3.5% in CCAB hybrids: most CCAB hybrid flowers were male-sterile. AABC and CCAB F1 hybrid plants averaged 5.5 and 0.5 seeds per plant, respectively, and BBAC F1 hybrids ~56 seeds/plant. In the second generation (S1), all confirmed self-pollinated progeny resulting from CCAB hybrids were sterile, producing no self-pollinated seeds. Three AABC S1 hybrids putatively resulting from unreduced gametes produced 3, 14, and 182 seeds each, while other AABC S1 hybrids averaged 1.5 seeds/plant (0–8). BBAC S1 hybrids averaged 44 seeds/plant (range 0–403). We also observed strong bias towards retention rather than loss of the haploid genomes, suggesting that the subgenomes in the Brassica allotetraploids are already highly interdependent, such that loss of one subgenome is detrimental to fertility and viability. Our results suggest that relationships between subgenomes determine hybridization outcomes in these species.