Distillation is an effective knowledge-transfer technique that uses predicted distributions of a powerful teacher model as soft targets to train a less-parameterized student model. A pre-trained high capacity teacher, however, is not always available. Recently proposed online variants use the aggregated intermediate predictions of multiple student models as targets to train each student model. Although group-derived targets give a good recipe for teacher-free distillation, group members are homogenized quickly with simple aggregation functions, leading to early saturated solutions. In this work, we propose Online Knowledge Distillation with Diverse peers (OKDDip), which performs two-level distillation during training with multiple auxiliary peers and one group leader. In the first-level distillation, each auxiliary peer holds an individual set of aggregation weights generated with an attention-based mechanism to derive its own targets from predictions of other auxiliary peers. Learning from distinct target distributions helps to boost peer diversity for effectiveness of group-based distillation. The second-level distillation is performed to transfer the knowledge in the ensemble of auxiliary peers further to the group leader, i.e., the model used for inference. Experimental results show that the proposed framework consistently gives better performance than state-of-the-art approaches without sacrificing training or inference complexity, demonstrating the effectiveness of the proposed two-level distillation framework.

No abstract

Age-related blood-brain barrier disruption and cerebromicrovascular rarefaction contribute importantly to the pathogenesis of both vascular cognitive impairment and dementia (VCID) and Alzheimer's disease (AD). Recent advances in geroscience research enable development of novel interventions to reverse age-related alterations of the cerebral microcirculation for prevention of VCID and AD. To facilitate this research there is an urgent need for sensitive and easy-to-adapt imaging methods, which enable longitudinal assessment of changes in BBB permeability and brain capillarization in aged mice, that could be used in vivo to evaluate treatment efficiency. To enable longitudinal assessment of changes in BBB permeability in aged mice equipped with a chronic cranial window, we adapted and optimized two different intravital two-photon imaging approaches. By assessing relative fluorescence changes over the baseline within a volume of brain tissue, after qualitative image subtraction of the brain microvasculature, we confirmed that in 24 month old C57BL/6J mice cumulative permeability of the microvessels to fluorescent tracers of different molecular weights (0.3 kDa to 40 kDa) is significantly increased as compared to that of 5 month old mice. Real-time recording of vessel cross-sections showed that apparent solute permeability of single microvessels is significantly increased in aged mice vs. young mice. Cortical capillary density, assessed both by intravital two-photon microscopy and optical coherence tomography (OCT) was also decreased in aged mice vs. young mice. The presented methods are optimized for longitudinal in vivo assessment of cerebromicrovascular health in preclinical geroscience research.

No abstract

Deep neural networks (DNNs) have shown serious vulnerability to adversarial examples with imperceptible perturbation to clean images. Most existing input-transformation based defense methods rely heavily on the learned external priors from an external large training dataset, while neglecting the rich image internal priors of the input itself, thus limiting the generalization of the defense models against the adversarial examples with biased image statistics from the external training dataset. Motivated by deep image prior that can capture rich image statistics from a single image, we propose an effective Deep Image Prior Driven Defense (DIPDefend) method against adversarial examples. With a DIP generator to fit the target/adversarial input, we find that our image reconstruction exhibits quite interesting learning preference from a feature learning perspectives, i.e., the early stage primarily learns the robust features resistant to adversarial perturbation, followed by learning non-robust features that are sensitive to adversarial perturbation. Besides, we develop an adaptive stopping strategy that adapts our method to diverse images. In this way, the proposed model obtains a unique defender for each individual adversarial input, thus being robust to various attackers. Experimental results demonstrate the superiority of our method over the state-of-the-art defense methods against white-box and black-box adversarial attacks. CCS CONCEPTS • Security and privacy → Trust frameworks; • Computing methodologies → Neural networks.