Aiding design and test optimization of analog circuits requires accurate models that can reliably capture complex dependencies of circuit performances on essential circuit and device parameters, and test signatures. We present a novel Bayesian learning technique, namely relevance vector and feature machine (RVFM), for characterizing analog circuits with sparse statistical regression models. RVFM not only produces accurate models learned from a moderate amount of simulation or measurement data, but also computes a probabilistically inferred weighting factor quantifying the criticality of each parameter as part of the overall learning framework, hence offering a powerful enabler for variability modeling, failure diagnosis, and test development. Compared to other popular learning-based techniques, the proposed RVFM produces more accurate models, requires less amount of training data, and extracts more reliable parametric ranking. The effectiveness of RVFM is demonstrated in terms of the statistical variability modeling of a low-dropout regulator (LDO) and the built-in self-test (BIST) development of a charge-pump phase-locked loop (PLL).