Early embryonic development is a complex process where undifferentiated cells lose their pluripotency and start to gastrulate. During gastrulation, three germ layers form, giving rise to different cell lineages and organs. This process is regulated by transcription factors and epigenetic regulators, including non-canonical polycomb repressive complex 1s (ncPRC1s). Previously, we reported that ncPRC1-member RYBP (RING1 and YY1 binding protein) is crucial for embryonic implantation and cardiac lineage commitment in mice. However, the role of RYBP in gastrulation and mesoderm formation has not yet been defined. In this study, we used 2D and 3D in vitro model systems, to analyze the role of RYBP in mesoderm formation. First, we showed that cardiac and endothelial progenitors–both derived from mesoderm–are underrepresented in the Rybp−/− cardiac colonies. In the absence of RYBP, the formation of major germ layers was also disrupted, and the expression of mesoderm- (Brachyury, Eomes, and Gsc) and endoderm-specific (Sox17, Gata4) genes was significantly downregulated. Using 3D embryoid bodies as gastrulation models, we showed that RYBP can co-localize with mesoderm lineage marker protein BRACHYURY and endoderm marker protein GATA4 and both proteins. In mutants, both proteins were detected at low levels and showed altered distribution. Additionally, we compared our in vitro results to available in vivo single-cell transcriptomes and showed that Rybp and Brachyury co-expressed in the primitive streak and six mesodermal clusters. Since caudal mesoderm exhibited one of the strongest co-expressions, we tested axial elongation in wt and Rybp−/− gastruloids. In the absence of RYBP, gastruloids exhibited shortened tails and low BRACHYURY levels in the tailbud. Finally, we identified BRACHYURY as a novel binding partner of RYBP and presented evidence of possible cooperative function during mesoderm formation and axial elongation. Together, our results demonstrate the previously unknown role of RYBP in mesoderm formation. We believe our findings will contribute to better understanding of the highly conserved process of gastrulation.