Transfer Learning for Bilingual Content Classification

Abstract

LinkedIn Groups provide a platform on which professionals with similar background, target and specialities can share content, take part in discussions and establish opinions on industry topics. As in most online social communities, spam content in LinkedIn Groups poses great challenges to the user experience and could eventually lead to substantial loss of active users. Building an intelligent and scalable spam detection system is highly desirable but faces difficulties such as lack of labeled training data, particularly for languages other than English. In this paper, we take the spam (Spanish) job posting detection as the target problem and build a generic machine learning pipeline for multi-lingual spam detection. The main components are feature generation and knowledge migration via transfer learning. Specifically, in the feature generation phase, a relatively large labeled data set is generated via machine translation. Together with a large set of unlabeled human written Spanish data, unigram features are generated based on the frequency. In the second phase, machine translated data are properly reweighted to capture the discrepancy from human written ones and classifiers can be built on top of them. To make effective use of a small portion of labeled data available in human written Spanish, an adaptive transfer learning algorithm is proposed to further improve the performance. We evaluate the proposed method on LinkedIn's production data and the promising results verify the efficacy of our proposed algorithm. The pipeline is ready for production.

WoLMIS: a labor market intelligence system for classifying web job vacancies

“…In [51] the authors developed a spam classification system for Spanish messages exchanged by LinkedIn users. The goal is to classify messages exchanged by social network users, messages are not necessarily job postings, e.g., they may be unsolicited advertisements.…”

Multiple‐components weights model for cross‐project software defect prediction

“…Every time the KMM is applied, it will utilise both component data and target data (whether they are labelled or not) to build a potential transferring bridge. In general, there is no way to infer a good estimator based on two different joint distributions in source component and target project because of the two distributions could be arbitrarily far apart [30]. Due to this unsolvable estimation problem, KMM assumes that the conditional probability of these two distributions are fixed, meaning that = .…”