Large-scale distributed training of deep neural networks suffer from the generalization gap caused by the increase in the effective mini-batch size. Previous approaches try to solve this problem by varying the learning rate and batch size over epochs and layers, or some ad hoc modification of the batch normalization. We propose an alternative approach using a second-order optimization method that shows similar generalization capability to first-order methods, but converges faster and can handle larger minibatches. To test our method on a benchmark where highly optimized first-order methods are available as references, we train ResNet-50 on ImageNet. We converged to 75% Top-1 validation accuracy in 35 epochs for mini-batch sizes under 16,384, and achieved 75% even with a mini-batch size of 131,072, which took only 978 iterations.

No abstract

Endoscopic submucosal dissection (ESD) is a safe and minimally invasive method for the treatment of early gastric cancer (EGC). However, whether ESD for EGC is also safe and feasible in patients aged ≥ 85 years is unclear. The patients enrolled in this study were divided into three groups: age ≥ 85 years (44 patients, 49 lesions), age 65–84 years (624 patients, 687 lesions), and age ≤ 64 years (162 patients, 174 lesions). We evaluated the incidence of adverse events (AEs) and overall survival (OS) and disease-specific survival (DSS). We analyzed the factors that had a significant impact on the prognosis of patients aged ≥ 85 years. No significant differences were found in the incidence of AEs among the three groups (p = 0.612). The OS was significantly lower in patients aged ≥ 85 years (p < 0.001). Conversely, DSS was not significantly worse in patients aged ≥ 85 years (p = 0.100). The poor Geriatric Nutritional Risk Index correlated with poor prognosis in patients aged ≥ 85 years (p < 0.001). ESD is a safe and valid treatment for EGC in patients aged ≥ 85 years. However, the indications should be carefully decided because it is difficult to estimate the survival contribution of ESD for EGC in patients aged ≥ 85 years, especially in those with poor nutritional status.

Large-scale distributed training of deep neural networks results in models with worse generalization performance as a result of the increase in the effective mini-batch size. Previous approaches attempt to address this problem by varying the learning rate and batch size over epochs and layers, or ad hoc modifications of batch normalization. We propose Scalable and Practical Natural Gradient Descent (SP-NGD), a principled approach for training models that allows them to attain similar generalization performance to models trained with first-order optimization methods, but with accelerated convergence. Furthermore, SP-NGD scales to large mini-batch sizes with a negligible computational overhead as compared to first-order methods. We evaluated SP-NGD on a benchmark task where highly optimized first-order methods are available as references: training a ResNet-50 model for image classification on ImageNet. We demonstrate convergence to a top-1 validation accuracy of 75.4% in 5.5 minutes using a mini-batch size of 32,768 with 1,024 GPUs, as well as an accuracy of 74.9% with an extremely large mini-batch size of 131,072 in 873 steps of SP-NGD.

No abstract