The capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost. * Equally major contributors † Work done as a member of the Google Brain Residency program (g.co/brainresidency)

Global warming is causing ocean warming and acidification. The distribution of Heliocidaris erythrogramma coincides with the eastern Australia climate change hot spot, where disproportionate warming makes marine biota particularly vulnerable to climate change. In keeping with near-future climate change scenarios, we determined the interactive effects of warming and acidification on fertilization and development of this echinoid. Experimental treatments (20-268C, pH 7.6-8.2) were tested in all combinations for the 'business-as-usual' scenario, with 208C/pH 8.2 being ambient. Percentage of fertilization was high (O89%) across all treatments. There was no difference in percentage of normal development in any pH treatment. In elevated temperature conditions, C48C reduced cleavage by 40 per cent and C68C by a further 20 per cent. Normal gastrulation fell below 4 per cent at C68C. At 268C, development was impaired. As the first study of interactive effects of temperature and pH on sea urchin development, we confirm the thermotolerance and pH resilience of fertilization and embryogenesis within predicted climate change scenarios, with negative effects at upper limits of ocean warming. Our findings place single stressor studies in context and emphasize the need for experiments that address ocean warming and acidification concurrently. Although ocean acidification research has focused on impaired calcification, embryos may not reach the skeletogenic stage in a warm ocean.

The most fragile skeletons produced by benthic marine calcifiers are those that larvae and juveniles make to support their bodies. Ocean warming, acidification, decreased carbonate saturation and their interactive effects are likely to impair skeletogenesis. Failure to produce skeleton in a changing ocean has negative implications for a diversity of marine species. We examined the interactive effects of warming and acidification on an abalone (Haliotis coccoradiata) and a sea urchin (Heliocidaris erythrogramma) reared from fertilization in temperature and pH/pCO 2 treatments in a climatically and regionally relevant setting. Exposure of ectodermal (abalone) and mesodermal (echinoid) calcifying systems to warming (þ28C to 48C) and acidification (pH 7.6 -7.8) resulted in unshelled larvae and abnormal juveniles. Haliotis development was most sensitive with no interaction between stressors. For Heliocidaris, the percentage of normal juveniles decreased in response to both stressors, although a þ28C warming diminished the negative effect of low pH. The number of spines produced decreased with increasing acidification/pCO 2 , and the interactive effect between stressors indicated that a þ28C warming reduced the negative effects of low pH. At þ48C, the developmental thermal tolerance was breached. Our results show that projected near-future climate change will have deleterious effects on development with differences in vulnerability in the two species.

Global climate change and ozone layer thinning will simultaneously expose organisms to increasingly stressful conditions. Early life stages of marine organisms, particularly eggs and larvae, are considered most vulnerable to environmental extremes. Here, we exposed encapsulated embryos of three common rocky shore gastropods to simultaneous combinations of ecologically realistic levels of ultraviolet radiation (UVR), water temperature stress and salinity stress to identify potential interactions and associated impacts of climate change. We detected synergistic effects with increases in mortality and retardation in development associated with the most physiologically stressful conditions. The effects of UVR were particularly marked, with mortality increasing up to 12-fold under stressful conditions. Importantly, the complex outcomes observed on applying multiple stressors could not have been predicted from examining environmental variables in isolation. Hence, we are probably dramatically underestimating the ecological impacts of climate change by failing to consider the complex interplay of combinations of environmental variables with organisms.

BackgroundAs the oceans simultaneously warm, acidify and increase in P CO2, prospects for marine biota are of concern. Calcifying species may find it difficult to produce their skeleton because ocean acidification decreases calcium carbonate saturation and accompanying hypercapnia suppresses metabolism. However, this may be buffered by enhanced growth and metabolism due to warming.Methodology/Principal FindingsWe examined the interactive effects of near-future ocean warming and increased acidification/P CO2 on larval development in the tropical sea urchin Tripneustes gratilla. Larvae were reared in multifactorial experiments in flow-through conditions in all combinations of three temperature and three pH/P CO2 treatments. Experiments were placed in the setting of projected near future conditions for SE Australia, a global change hot spot. Increased acidity/P CO2 and decreased carbonate mineral saturation significantly reduced larval growth resulting in decreased skeletal length. Increased temperature (+3°C) stimulated growth, producing significantly bigger larvae across all pH/P CO2 treatments up to a thermal threshold (+6°C). Increased acidity (-0.3-0.5 pH units) and hypercapnia significantly reduced larval calcification. A +3°C warming diminished the negative effects of acidification and hypercapnia on larval growth.Conclusions and SignificanceThis study of the effects of ocean warming and CO2 driven acidification on development and calcification of marine invertebrate larvae reared in experimental conditions from the outset of development (fertilization) shows the positive and negative effects of these stressors. In simultaneous exposure to stressors the dwarfing effects of acidification were dominant. Reduction in size of sea urchin larvae in a high P CO2 ocean would likely impair their performance with negative consequent effects for benthic adult populations.