Identifying Similar Question Pairs Using Machine Learning Techniques

Abstract

Background/Objectives: Every day millions of people visit search engines like Quora, reedit, stack overflow, etc., the demand for new intelligent techniques is growing, to help individuals find better solutions. Methods: In our proposed system, the Quora datasets were filtered using SQLite which takes one-quarter of the time taken to pre-process the same dataset using existing approaches like python functions. We used machine learning techniques namely the Random Forest, Logistic Regression, Linear SVM (Support Vector Machine) and XGBoost to analyze and identify the most suitable model. Findings: The error log loss functions (0.887, 0.521, 0.654 and 0.357) of the above machine learning techniques were analyzed and compared respectively. The performance of XGBoost is the best among the other models, hence XGBoost is the most efficient model. Conclusion/Future Scope: It is concluded that XGBoost has outperformed other machine learning techniques discussed in the study. It is also found that pre-processing using SQLite has improved the response time. In the future, we would like to apply a similar approach for various other search engines that are available like reedit, stack overflow, etc. and one could ensemble the best models of each type (linear, tree-based, and neural network).

Enhancing Question Pairs Identification with Ensemble Learning: Integrating Machine Learning and Deep Learning Models

“…The Gradient Boosting Machine and AdaBoost models achieved the highest train accuracies, outperforming the Decision Tree, Random Forest, and Extra Trees models. These results are in agreement with authors of [17] research, where their goal was to find the best machine learning technique for removing all duplicate questions and increasing user satisfaction. Using a real-time dataset, this work trained and tested four machine learning models to recognize duplicate inquiries.…”

“…Anishaa et al [17] proposed a novel approach by filtration of the Quora datasets using SQLite which takes one-quarter the time it takes to pre-process the same dataset using existing methodologies such as python functions. It concluded that XGBoost outperformed the other machine learning approaches discussed, it has also been discovered that pre-processing with SQLite has improved response time.…”

Strengthening Sentence Similarity Identification Through OpenAI Embeddings and Deep Learning

“…After execution, the random, logistic regression, linear SVM, and XGBoost error parameters referred to from the log loss function are found to be, respectively, 0.887, 0.521, 0.654, and 0.357. As a result of the unique pre-processing activities carried out using PL/SQL, which improve response time overall, the result demonstrates that XGBoost is the best model, delivering the greatest accuracy in the shortest period of time [9].…”