In this paper, the problem of optimal gradient lossless compression in Deep Neural Network (DNN) training is considered. Gradient compression is relevant in many distributed DNN training scenarios, including the recently popular federated learning (FL) scenario in which each remote users are connected to the parameter server (PS) through a noiseless but rate limited channel. In distributed DNN training, if the underlying gradient distribution is available, classical lossless compression approaches can be used to reduce the number of bits required for communicating the gradient entries. Mean field analysis has suggested that gradient updates can be considered as independent random variables, while Laplace approximation can be used to argue that gradient has a distribution approximating the normal (Norm) distribution in some regimes. In this paper we argue that, for some networks of practical interest, the gradient entries can be well modelled as having a generalized normal (GenNorm) distribution. We provide numerical evaluations to validate that the hypothesis GenNorm modelling provides a more accurate prediction of the DNN gradient tail distribution. Additionally, this modeling choice provides concrete improvement in terms of lossless compression of the gradients when applying classical fixto-variable lossless coding algorithms, such as Huffman coding, to the quantized gradient updates. This latter results indeed provides an effective compression strategy with low memory and computational complexity that has great practical relevance in distributed DNN training scenarios.
Indirectly driven inertial confinement fusion implosions using a three-step- shaped pulse were performed at the 100kJ laser facility. At late time of the pulse, deposition of laser energy and distribution of x-ray radiation were significantly disturbed by motion of gold plasma in the original gas-filled cylindrical hohlraum with gold wall. As a result, the lack of x-ray drive at the equator of the capsule generated an unacceptable oblate implosion. In the I-raum modified from the above cylindrical hohlraum, the initial positions of outer laser spots and gold bubbles were properly shifted to modify the disturbed radiation distribution, due to plasma evolution, resulting in a spherically symmetric drive on the capsule. In implosion shots with almost the same drive pulse, owing to improved symmetry, an spherical hotspot was observed in the new I-raum, and YOS (measured neutron yield over simulated one dimensionally ) was up to 30%, while an oblate hotspot was observed in the cylinder, and YOS was 13%. Both simulation calculations and experimental measurements show that, the I-raum can be used to significantly reduce the impact of gold bubble expansion in the three-step-shaped pulse driven implosion, which helps to tune the drive and implosion symmetry, and to improve its over-all performance.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.