Parameter optimization for machine-learning of word sense disambiguation

Abstract

Link to this article: http://journals.cambridge.org/abstract_S1351324902003005How to cite this article: V. HOSTE, I. HENDRICKX, W. DAELEMANS and A. VAN DEN BOSCH (2002). Parameter optimization for machine-learning of word sense disambiguation. Natural Language Engineering, 8, pp 311-325 AbstractVarious Machine Learning (ML) approaches have been demonstrated to produce relatively successful Word Sense Disambiguation (WSD) systems. There are still unexplained differences among the performance measurements of different algorithms, hence it is warranted to deepen the investigation into which algorithm has the right 'bias' for this task. In this paper, we show that this is not easy to accomplish, due to intricate interactions between information sources, parameter settings, and properties of the training data. We investigate the impact of parameter optimization on generalization accuracy in a memory-based learning approach to English and Dutch WSD. A 'word-expert' architecture was adopted, yielding a set of classifiers, each specialized in one single wordform. The experts consist of multiple memorybased learning classifiers, each taking different information sources as input, combined in a voting scheme. We optimized the architectural and parametric settings for each individual word-expert by performing cross-validation experiments on the learning material. The results of these experiments show that the variation of both the algorithmic parameters and the information sources available to the classifiers leads to large fluctuations in accuracy. We demonstrate that optimization per word-expert leads to an overall significant improvement in the generalization accuracies of the produced WSD systems.

Effects of information and machine learning algorithms on word sense disambiguation with small datasets

“…Such relations have been reported in [5]. For example, if the baseline ambiguity were high (e.g., 95/100), it would be hard for any algorithm to learn to correctly classify the additional five cases.…”

“…This majority sense performance, also called lexical default [5], served as the baseline for our study. We choose 65% because others have found that high majority sense results in a very skewed dataset that provides insufficient examples to automatically learn from [3].…”

“…In addition to different algorithms, the amount, relevance, and precision of the information affect performance. Hoste et al [5] manipulated the feature information and algorithm settings in an extensive set of studies. They argued that algorithms with default settings do not provide a sufficient base for comparison.…”

“…They use WordNet as their knowledge source and calculate statistics for the most probable senses using the Internet as a corpus. Hoste et al [5] describe a memory-based learning approach where the algorithm keeps all training examples in memory. A classification decision is made based on similarity between new input and stored examples.…”

“…For example, Mooney [4] used 300, 600, and 1200 examples for training and showed that performance increased with more examples. Hoste et al [5] argues that increasing the size of the gold standard, e.g., by a factor of 1000, has more effect on performance than individual algorithm biases. Alas, compiling such gold standards is time-consuming and difficult.…”