We introduce exBERT, a training method to extend BERT pre-trained models from a general domain to a new pre-trained model for a specific domain with a new additive vocabulary under constrained training resources (i.e., constrained computation and data). exBERT uses a small extension module to learn to adapt an augmenting embedding for the new domain in the context of the original BERT's embedding of a general vocabulary. The exBERT training method is novel in learning the new vocabulary and the extension module while keeping the weights of the original BERT model fixed, resulting in a substantial reduction in required training resources. We pre-train exBERT with biomedical articles from ClinicalKey and PubMed Central, and study its performance on biomedical downstream benchmark tasks using the MTL-Bioinformatics-2016 dataset. We demonstrate that exBERT consistently outperforms prior approaches when using limited corpus and pretraining computation resources.

No abstract

No abstract

Compressive sensing, which enables signal recovery from fewer samples than traditional sampling theory dictates, assumes that the sampling process is linear. However, this linearity assumption may not hold in the analog domain without significant trade-offs, such as power amplifiers sacrificing substantial power efficiency in exchange for producing linear outputs. Since compressive sensing is most impactful when implemented in the analog domain, it is of interest to integrate the nonlinearity in compressive measurements into the signal recovery process such that nonlinear effects can be mitigated. As such, in this paper, we describe a nonlinear compressive sensing formulation and associated signal recovery algorithms, providing both compression and improved efficiency of a power amplifier simultaneously with one procedure. We present evaluations of the proposed framework using both measurements from real power amplifiers and simulations.

means clustering: Automatic parameter search and distributed computing implementation."