Shrinkage learning to improve SVM with hints

Abstract

The Support Vector Machine (SVM) is one of the most effective and used algorithms, when targeting classification. Despite its large success, SVM is mainly afflicted by two issues: (i) some hyperparameters must be tuned in advance and are, in practice, identified through computationally intensive procedures; (ii) possible a-priori knowledge about the problem (e.g. doctor expertise in medical applications) cannot be straightforwardly exploited. In this paper, we introduce a new approach, able to cope with the two previous problems: several experiments, performed on real-world benchmarking datasets, show that our method outperforms, on average, other techniques proposed in the literature

Towards Knowledgeable Supervised Lifelong Learning Systems

“…Transfer is commonly performed from a set of previous models or hypotheses, rather than from a single source. A number of theoretical (Kuzborskij & Orabona, 2013;Kuzborskij, 2018), experimental (Yang et al, 2007;Aytar & Zisserman, 2011;Tommasi et al, 2014;Kuzborskij et al, 2015;Oneto et al, 2015;Mozafari & Jamzad, 2016;Wang & Hebert, 2016), and application-specific methods (Valerio, Passarella, & Conti, 2016) have been proposed. Transferring from previous hypotheses rather than from previous data as in classic transfer learning provides a potential for long-term learning systems, since these hypotheses are the only knowledge that needs to be stored to aid learning of future tasks.…”

“…Hypothesis transfer learning is an alternative for transferring knowledge from previous models or hypotheses learned during previous tasks. Most research in hypothesis transfer learning with SVM has proposed to transfer knowledge from source hypotheses as a whole, without distinguishing fragments of knowledge that can be potentially more useful for a target task (Yang et al, 2007;Aytar & Zisserman, 2011;Tommasi et al, 2014;Kuzborskij et al, 2015;Oneto et al, 2015;Mozafari & Jamzad, 2016;Wang & Hebert, 2016). Furthermore, most existing solutions require the target hypothesis to be represented in terms of both the learned set of parameters and the existing source hypotheses.…”

“…Shrinkage Learning SVM, SL-SVM (Oneto et al, 2015), was proposed as a method aimed to consider a-priori knowledge for training a target SVM hypothesis. The method proposed the use of hints to initialize a SVM hypothesis to be learned.…”

“…An example of a hypothesis is a function f of the form Y = wx + b, that can be used to classify an example x from a sample X represented in a feature space X into a sample of classes or labels Y from the class or label space Y, using a learned set of weights w and a bias term b. Support Vector Machines (SVM) has been one of the recent avenues of research in hypothesis transfer learning (Yang, Yan, & Hauptmann, 2007;Tommasi, Orabona, & Caputo, 2014;Kuzborskij, Orabona, & Caputo, 2015;Oneto, Ghio, Ridella, & Anguita, 2015;Mozafari & Jamzad, 2016;Wang & Hebert, 2016). SVM is a theoretically grounded technique used for both classification and regression.…”

“…ν-SVM (Schölkopf, Smola, Williamson, & Bartlett, 2000) and learning with privileged information (Vapnik & Vashist, 2009), which raises the potential to combine these variants to simultaneously achieve different properties of learning. Third, a variety of studies have demonstrated the potential use of additional information to improve learning of an SVM hypothesis (Schölkopf, Burges, & Vapnik, 1996;Decoste & Schölkopf, 2002;Oneto et al, 2015;Vapnik & Izmailov, 2015). A recent inter-est has also arisen for supervised lifelong learning research using SVM (Fei, Wang, & Liu, 2016).…”