No abstract

AI is undergoing a paradigm shift with the rise of models (e.g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks. We call these models foundation models to underscore their critically central yet incomplete character. This report provides a thorough account of the opportunities and risks of foundation models, ranging from their capabilities (e.g., language, vision, robotics, reasoning, human interaction) and technical principles (e.g., model architectures, training procedures, data, systems, security, evaluation, theory) to their applications (e.g., law, healthcare, education) and societal impact (e.g., inequity, misuse, economic and environmental impact, legal and ethical considerations). Though foundation models are based on standard deep learning and transfer learning, their scale results in new emergent capabilities, and their effectiveness across so many tasks incentivizes homogenization. Homogenization provides powerful leverage but demands caution, as the defects of the foundation model are inherited by all the adapted models downstream. Despite the impending widespread deployment of foundation models, we currently lack a clear understanding of how they work, when they fail, and what they are even capable of due to their emergent properties. To tackle these questions, we believe much of the critical research on foundation models will require deep interdisciplinary collaboration commensurate with their fundamentally sociotechnical nature.

SUMMARY Stem-cell differentiation to desired lineages requires navigating alternating developmental paths often leading to unwanted cell-types. Hence comprehensive developmental roadmaps are crucial to channel stem-cell differentiation towards desired fates. To this end, here we map bifurcating lineage choices leading from pluripotency to twelve human mesodermal lineages, including bone, muscle and heart. We defined the extrinsic signals controlling each binary lineage decision, enabling us to logically block differentiation towards unwanted fates and rapidly steer pluripotent stem cells towards 80–99% pure human mesodermal lineages at most branchpoints. This strategy enabled the generation of human bone and heart progenitors that could engraft in respective in vivo models. Mapping stepwise chromatin and single-cell gene expression changes in mesoderm development uncovered somite segmentation, a previously-unobservable human embryonic event transiently marked by HOPX expression. Collectively this roadmap enables navigation of mesodermal development to produce transplantable human tissue progenitors and uncover developmental processes.

Overparameterized neural networks can be highly accurate on average on an i.i.d.test set yet consistently fail on atypical groups of the data (e.g., by learning spurious correlations that hold on average but not in such groups). Distributionally robust optimization (DRO) allows us to learn models that instead minimize the worst-case training loss over a set of pre-defined groups. However, we find that naively applying group DRO to overparameterized neural networks fails: these models can perfectly fit the training data, and any model with vanishing average training loss also already has vanishing worst-case training loss. Instead, their poor worst-case performance arises from poor generalization on some groups. By coupling group DRO models with increased regularization-stronger-than-typical 2 regularization or early stopping-we achieve substantially higher worst-group accuracies, with 10-40 percentage point improvements on a natural language inference task and two image tasks, while maintaining high average accuracies. Our results suggest that regularization is critical for worst-group generalization in the overparameterized regime, even if it is not needed for average generalization. Finally, we introduce and give convergence guarantees for a stochastic optimizer for the group DRO setting, underpinning the empirical study above.

No abstract