Pre-training and fine-tuning have achieved remarkable success in many downstream natural language processing (NLP) tasks. Recently, pre-training methods tailored for information retrieval (IR) have also been explored, and the latest success is the PROP method which has reached new SOTA on a variety of ad-hoc retrieval benchmarks. The basic idea of PROP is to construct the representative words prediction (ROP) task for pre-training inspired by the query likelihood model. Despite its exciting performance, the effectiveness of PROP might be bounded by the classical unigram language model adopted in the ROP task construction process. To tackle this problem, we propose a bootstrapped pre-training method (namely B-PROP) based on BERT for ad-hoc retrieval. The key idea is to use the powerful contextual language model BERT to replace the classical unigram language model for the ROP task construction, and re-train BERT itself towards the tailored objective for IR. Specifically, we introduce a novel contrastive method, inspired by the divergence-from-randomness idea, to leverage BERT's self-attention mechanism to sample representative words from the document. By further fine-tuning on downstream ad-hoc retrieval tasks, our method achieves significant improvements over baselines without pre-training or with other pre-training methods, and further pushes forward the SOTA on a variety of ad-hoc retrieval tasks.

Conventional artificial intelligence (AI) machine vision technology, based on the von Neumann architecture, uses separate sensing, computing, and storage units to process huge amounts of vision data generated in sensory terminals. The frequent movement of redundant data between sensors, processors and memory, however, results in high-power consumption and latency. A more efficient approach is to offload some of the memory and computational tasks to sensor elements that can perceive and process the optical signal simultaneously. Here, we proposed a non-volatile photomemristor, in which the reconfigurable responsivity can be modulated by the charge and/or photon flux through it and further stored in the device. The non-volatile photomemristor has a simple two-terminal architecture, in which photoexcited carriers and oxygen-related ions are coupled, leading to a displaced and pinched hysteresis in the current-voltage characteristics. For the first time, non-volatile photomemristors implement computationally complete logic with photoresponse-stateful operations, for which the same photomemristor serves as both a logic gate and memory, using photoresponse as a physical state variable instead of light, voltage and memresistance. The polarity reversal of photomemristors shows great potential for in-memory sensing and computing with feature extraction and image recognition for neuromorphic vision.

No abstract