Adaptive regularization of weight vectors

Crammer, Koby; Kulesza, Alex; Dredze, Mark

doi:10.1007/s10994-013-5327-x

Cited by 225 publications

(256 citation statements)

References 24 publications

Supporting

Mentioning

253

Contrasting

Unclassified

Order By: Relevance

“…Instead, online learning is commonly used, since it updates training parameters one by one for each of the training samples and requires only one sample at a time. In ILSVRC2012, most of the team used SGD or Passive Aggressive (PA) as an online learning method, while we use AROW [2].…”

Section: Linear Classifiermentioning

confidence: 99%

“…AROW [2] is an online learning method of linear classifiers, which is robust to noise label. This property is suitable for a large-scale data, especially data gathered from the Web.…”

Section: Adaptive Regularization Of Weights (Arow)mentioning

confidence: 99%

“…[10]. We use AROW [2] for an online learning method of linear classifiers, while they used Stochastic Gradient Descent (SGD).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

ILSVRC on a Smartphone

Kawano

Yanai

2014

IPSJ Transactions on Computer Vision and Applications

View full text Add to dashboard Cite

Abstract:In this work, to the best of our knowledge, we propose a stand-alone large-scale image classification system running on an Android smartphone. The objective of this work is to prove that mobile large-scale image classification requires no communication to external servers. To do that, we propose a scalar-based compression method for weight vectors of linear classifiers. As an additional characteristic, the proposed method does not need to uncompress the compressed vectors for evaluation of the classifiers, which brings the saving of recognition time. We have implemented a large-scale image classification system on an Android smartphone, which can perform 1000-class classification for a given image in 0.270 seconds. In the experiment, we show that compressing the weights to 1/8 leaded to only 0.80% performance loss for 1000-class classification with the ILSVRC2012 dataset. In addition, the experimental results indicate that weight vectors compressed in low bits, even in the binarized case (bit = 1), are still valid for classification of high dimensional vectors.

show abstract

Section: Linear Classifiermentioning

confidence: 99%

“…AROW [2] is an online learning method of linear classifiers, which is robust to noise label. This property is suitable for a large-scale data, especially data gathered from the Web.…”

Section: Adaptive Regularization Of Weights (Arow)mentioning

confidence: 99%

See 1 more Smart Citation

ILSVRC on a Smartphone

Kawano

Yanai

2014

IPSJ Transactions on Computer Vision and Applications

View full text Add to dashboard Cite

show abstract

“…These filters are reminiscent of the celebrated Kalman filter [18], which was motivated and analyzed in a stochastic setting with Gaussian noise. Finally, few second-order algorithms were recently proposed in other contexts [6,8,11,20].…”

Section: Related Workmentioning

confidence: 99%

Weighted Last-Step Min-Max Algorithm with Improved Sub-logarithmic Regret

Moroshko

Crammer

2012

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

In online learning the performance of an algorithm is typically compared to the performance of a fixed function from some class, with a quantity called regret. Forster [12] proposed a last-step min-max algorithm which was somewhat simpler than the algorithm of Vovk [26], yet with the same regret. In fact the algorithm he analyzed assumed that the choices of the adversary are bounded, yielding artificially only the two extreme cases. We fix this problem by weighing the examples in such a way that the min-max problem will be well defined, and provide analysis with logarithmic regret that may have better multiplicative factor than both bounds of Forster [12] and Vovk [26]. We also derive a new bound that may be sub-logarithmic, as a recent bound of Orabona et.al [21], but may have better multiplicative factor. Finally, we analyze the algorithm in a weak-type of non-stationary setting, and show a bound that is sublinear if the non-stationarity is sub-linear as well.

show abstract

“…Adaptive Regularization of Weight Vectors (AROW) [18] is another online discriminative training method for binary classification that has been proposed as an approach to resolve overfitting. This is achieved by gradually learning parameters to correctly classify the training data, without guaranteeing that the current example is correctly classified.…”

Section: Introductionmentioning

confidence: 99%

Structured Adaptive Regularization of Weight Vectors for a Robust Grapheme-to-Phoneme Conversion Model

Kubo

Sakti

Neubig

et al. 2014

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

SUMMARYGrapheme-to-phoneme (g2p) conversion, used to estimate the pronunciations of out-of-vocabulary (OOV) words, is a highly important part of recognition systems, as well as text-to-speech systems. The current state-of-the-art approach in g2p conversion is structured learning based on the Margin Infused Relaxed Algorithm (MIRA), which is an online discriminative training method for multiclass classification. However, it is known that the aggressive weight update method of MIRA is prone to overfitting, even if the current example is an outlier or noisy. Adaptive Regularization of Weight Vectors (AROW) has been proposed to resolve this problem for binary classification. In addition, AROW's update rule is simpler and more efficient than that of MIRA, allowing for more efficient training. Although AROW has these advantages, it has not been applied to g2p conversion yet. In this paper, we first apply AROW on g2p conversion task which is structured learning problem. In an evaluation that employed a dataset generated from the collective knowledge on the Web, our proposed approach achieves a 6.8% error reduction rate compared to MIRA in terms of phoneme error rate. Also the learning time of our proposed approach was shorter than that of MIRA in almost datasets. key words: g2p conversion, out-of-vocabulary word, online discriminative training, structured learning, AROW

show abstract

Adaptive regularization of weight vectors

Cited by 225 publications

References 24 publications

ILSVRC on a Smartphone

ILSVRC on a Smartphone

Weighted Last-Step Min-Max Algorithm with Improved Sub-logarithmic Regret

Structured Adaptive Regularization of Weight Vectors for a Robust Grapheme-to-Phoneme Conversion Model

Contact Info

Product

Resources

About