Assessing fertilized human embryos is crucial for in vitro-fertilization (IVF), a task being revolutionized by artificial intelligence and deep learning. Existing models used for embryo quality assessment and chromosomal abnormality (ploidy) detection could be significantly improved by effectively utilizing time-lapse imaging to identify critical developmental time points for maximizing prediction accuracy. Addressing this, we developed and compared various embryo ploidy status prediction models across distinct embryo development stages. We present BELA (Blastocyst Evaluation Learning Algorithm), a state-of-the-art ploidy prediction model surpassing previous image- and video-based models, without necessitating subjective input from embryologists. BELA uses multitask learning to predict quality scores that are used downstream to predict ploidy status. By achieving an AUC of 0.76 for discriminating between euploidy and aneuploidy embryos on the Weill Cornell dataset, BELA matches the performance of models trained on embryologists’ manual scores. While not a replacement for preimplantation genetic testing for aneuploidy (PGT-A), BELA exemplifies how such models can streamline the embryo evaluation process, reducing time and effort required by embryologists.