Germ cells are the origin of new individuals. Hence, specifying germ cell identity is crucial for reproduction. The recent establishment of in vitro culture systems for generating oocytes from mouse pluripotent stem cells provides a basis for progress in studies of oogenesis and reproductive technology. However, currently the developmental competence of in vitro generated oocytes is low compared to in vivo grown oocytes. The causes underlying poor oocyte quality remain to be determined. By reconstituting germ cell development in culture from different developmental starting points within gametogenesis, we show that the differentiation of primordial germ cells (PGCs) and primordial germ cell-like cells (PGCLCs) to growing oocytes (GROs), as well as the subsequent growth of follicles are critical culture steps for specifying competence of fully-grown oocytes (FGOs) for preimplantation development. A systematic comparison of transcriptomes of single oocytes having undergone different in vitro culture trajectories identifies genes normally upregulated during oocyte growth to be susceptible for mis-regulation during in vitro oogenesis. Many of such genes have been described as targets of Polycomb repressive complexes (PRCs). Deregulation of Polycomb repression therefore likely perturbs the accumulation of cytoplasmic factors and/or setting of chromatin states in FGOs that are required for embryonic development after fertilization. Conversely, in vitro derived oocytes often displayed failure of zygotic genome activation (ZGA) and abnormal acquisition of 5-hydroxymethylcytosine (5hmC) on maternal chromosomes after activation. In addition, subcellular delocalization of pyruvate dehydrogenase (PDH) and of STELLA were observed suggesting new molecular markers for defective oocyte development. Our study identifies epigenetic regulation at an early stage of oogenesis as crucial for developmental competence and suggests specific in vitro culture steps as targets for improving oocyte quality.