The Quora Question Pair Similarity Problem
Quora问题对相似度问题
The Quora Question Pair Similarity Problem
Quora问题对相似度问题
A beginner’s journey through the various life cycles of a problem on Kaggle.
一个初学者在Kaggle上经历问题各个生命周期的旅程。
This is my first case study, so you can expect a beginner-friendly data analysis and model building. I have used only the classical machine learning models for this problem. However, working on this case study was a great learning experience for me. And in this blog, I will try to share with you as much as possible.
这是我的第一个案例研究，所以你可以期待一个对初学者友好的数据分析和模型构建。我只使用经典的机器学习模型来解决这个问题。然而，做这个案例研究对我来说是一次很棒的学习经历。在这篇博客中，我会尽可能多地与大家分享。
In the blog, I will write only the summary. You can view the full notebook here and you can view the code on github.
在博客中，我只写摘要。你可以在这里查看完整的notebook，也可以在这里查看GitHub上的代码。
And to all the experienced folks out there, I would love your feedback for future case studies. ??
对于所有有经验的朋友，我非常期待你们对未来案例研究的反馈。
Table of contents
目录
Introduction
引言
Business Objectives and Constraints
业务目标和约束
Data Overview
数据概览
Business Metrics
业务指标
Basic EDA
基础EDA
Data Cleaning
数据清洗
Feature Extraction
特征提取
EDA with Features
基于特征的EDA
Featurization with SentenceBERT
使用SentenceBERT进行特征化
i. EDA on new features related to SentenceBERT
i. 基于SentenceBERT相关新特征的EDA
Data Pre-processing
数据预处理
Training Models
模型训练
i. Support Vector Classifier
i.支持向量分类器
ii. Random Forest
II.随机森林
iii. XGBoost
iii. XGBoost
iv. Another XGBoost ?
iv. 另一个XGBoost
Final Thoughts
最终思考
References
参考文献
Introduction
引言
Quora is a platform for Q&A, just like StackOverflow. But quora is more of a general-purpose Q&A platform that means there is not much code like in StackOverflow.
Quora是一个Q&A平台，就像StackOverflow一样。但Quora更像是一个通用型的Q&A平台，这意味着它不像StackOverflow那样有很多代码相关内容。
One of the many problems that quora face is the duplication of questions. Duplication of question ruins the experience for both the questioner and the answerer. Since the questioner is asking a duplicate question, we can just show him/her the answers to the previous question. And the answerer doesn’t have to repeat his/her answer for essentially the same questions.
Quora面临的众多问题之一就是问题的重复。问题重复会破坏提问者和回答者双方的体验。因为提问者问的是一个重复的问题，我们可以直接向他/她展示之前问题的答案。而回答者也不必为本质相同的问题重复作答。
For example, we have a question like “How can I be a good geologist?” and there are some answers to that question. Later someone else asks another question like “What should I do to be a great geologist?”.
例如，我们有一个问题像"怎样才能成为一名好的地质学家？"，并且这个问题有一些答案。后来有人问另一个问题，比如"我应该怎么做才能成为一名优秀的地质学家？"。
We can see that both the questions are asking the same thing. Even though the wordings for the question are different, the intention of both questions is the same.
我们可以看到这两个问题问的是同一件事。尽管问题的措辞不同，但两个问题的意图是相同的。
So the answers will be the same for both questions. That means we can just show the answers to the first question. That way the person who is asking the question will get the answers immediately and people who have answered already the first question don’t have to repeat themselves.
所以两个问题的答案将是相同的。这意味着我们只需展示第一个问题的答案即可。这样，提问者可以立即得到答案，而已经回答过第一个问题的人也不必再重复。
This problem is available on Kaggle as a competition. https://www.kaggle.com/c/quora-question-pairs
这个问题在Kaggle上作为一个竞赛项目提供。https://www.kaggle.com/c/quora-question-pairs
So given two questions, our main objective is to find whether they are similar. So let’s do some magic with ML. ?
所以，给定两个问题，我们的主要目标是判断它们是否相似。让我们用机器学习来做一些神奇的事情吧。
Business Objectives and Constraints
业务目标与约束
There is no strict latency requirement.
没有严格的延迟要求。
We would like to have interpretability but it is not absolutely mandatory.
我们希望有一定的可解释性，但这并不是绝对必要的。
The cost of misclassification is medium.
误分类的成本中等。
Both classes (duplicate or not) are equally important.
两个类别（重复或不重复）同等重要。
Data Overview
数据概览
Available Columns: id, qid1, qid2, question1, question2, is_duplicate
可用列：id, qid1, qid2, question1, question2, is_duplicate
Class labels: 0, 1
类别标签：0， 1
Total training data / No. of rows: 404290
训练数据总量/行数：404290
No. of columns: 6
列数：6
is_duplicate is the dependent variable.
is_duplicate是因变量。
No. of non-duplicate data points is 255027
非重复数据点数量：255027
No. of duplicate data points is 149263
重复数据点数量：149263
We have 404290 training data points. And only 36.92% are positive. That means it is an imbalanced dataset.
我们有404290个训练数据点。其中只有36.92%是正例。这意味着这是一个不平衡数据集。
Business Metrics
业务指标
It is a binary classification.
这是一个二分类问题。
We need to minimize the log loss for this challenge.
本次挑战我们需要最小化对数损失。
Basic EDA
基本EDA
Test data don’t have question ids. So the independent variables are question1, question2 and the dependent variable is is_duplicate.
测试数据没有问题ID。所以自变量是question1和question2，因变量是is_duplicate。
3 rows had null values. So We removed them and now We have 404287 question pairs for training.
有3行存在空值。所以我们删除了它们，现在我们有404287对问题对用于训练。
36.92% of question pairs are duplicates and 63.08% of questions pair non-duplicate.
36.92%的问题对是重复的，63.08%的问题对是非重复的。
Out of 808574 total questions (including both question1 and question2), 537929 are unique.
在808574个总问题（包括question1和question2）中，有537929个是唯一的。
Most of the questions are repeated very few times. Only a few of them are repeated multiple times.
大多数问题重复次数很少。只有少数问题被重复多次。
One question is repeated 157 times which is the max number of repetitions.
有一个问题被重复了157次，这是最大的重复次数。
There are some questions with very few characters, which does not make sense. It will be taken care of later with Data Cleaning.
有些问题只有很少的字符，这没有意义。这将在后面的数据清洗中处理。
Data Cleaning
数据清洗
We have converted everything to lower case.
我们将所有内容转换为小写。
We have removed contractions.
我们移除了缩略词。
We have replaced currency symbols with currency names.
我们将货币符号替换为货币名称。
We have also removed hyperlinks.
我们还移除了超链接。
We have removed non-alphanumeric characters.
我们移除了非字母数字字符。
We have removed inflections with word lemmatizer.
我们使用词形还原移除了词形变化。
We have also removed HTML tags.
我们还移除了HTML标签。
Feature Extraction
特征提取
We have created 23 features from the questions.
我们从问题中创建了23个特征。
We have created features q1_char_num, q2_char_num with count of characters for both questions.
我们创建了q1_char_num、q2_char_num特征，用于统计两个问题的字符数。
We have created features q1_word_num, q2_word_num with count of characters for both questions.
我们创建了q1_word_num、q2_word_num特征，用于统计两个问题的单词数。
We have created total_word_num feature which is equal to sum of q1_word_num and q2_word_num.
我们创建了total_word_num特征，等于q1_word_num和q2_word_num之和。
We have created differ_word_num feature which is absolute difference between q1_word_num and q2_word_num.
我们创建了differ_word_num特征，即q1_word_num和q2_word_num之间的绝对差。
We have created same_first_word feature which is 1 if both questions have same first word otherwise 0.
我们创建了same_first_word特征，如果两个问题的第一个单词相同则为1，否则为0。
We have created same_last_word feature which is 1 if both questions have same last word otherwise 0.
我们创建了same_last_word特征，如果两个问题的最后一个单词相同则为1，否则为0。
We have created total_unique_word_num feature which is equal to total number of unique words in both questions.
我们创建了total_unique_word_num特征，等于两个问题中唯一单词的总数。
We have created total_unique_word_withoutstopword_num feature which is equal to total number of unique words in both questions without the stop words.
我们创建了total_unique_word_withoutstopword_num特征，等于两个问题中不含停用词的唯一单词总数。
The total_unique_word_num_ratio is equal to total_unique_word_num divided by total_word_num.
total_unique_word_num_ratio等于total_unique_word_num除以total_word_num。
We have created common_word_num feature which is count of total common words in both questions.
我们创建了common_word_num特征，统计两个问题中共同单词的总数。
The common_word_ratio feature is equal to common_word_num divided by total_unique_word_num.
common_word_ratio特征等于common_word_num除以total_unique_word_num。
The common_word_ratio_min is equal to common_word_num divided by minimum number of words between question 1 and question 2.
common_word_ratio_min等于common_word_num除以问题1和问题2之间单词数量的最小值。
The common_word_ratio_max is equal to common_word_num divided by maximum number of words between question 1 and question 2.
common_word_ratio_max等于common_word_num除以问题1和问题2之间单词数量的最大值。
We have created common_word_withoutstopword_num feature which is count of total common words in both questions excluding the stopwords.
我们创建了common_word_withoutstopword_num特征，统计两个问题中不含停用词的共同单词总数。
The common_word_withoutstopword_ratio feature is equal to common_word_withoutstopword_num divided by total_unique_word_withoutstopword_num.
common_word_withoutstopword_ratio特征等于common_word_withoutstopword_num除以total_unique_word_withoutstopword_num。
The common_word_withoutstopword_ratio_min is equal to common_word_withoutstopword_num divided by minimum number of words between question 1 and question 2 excluding the stopwords.
common_word_withoutstopword_ratio_min等于common_word_withoutstopword_num除以问题1和问题2之间不含停用词的单词数量的最小值。
The common_word_withoutstopword_ratio_max is equal to common_word_withoutstopword_num divided by maximum number of words between question 1 and question 2 excluding the stopwords.
common_word_withoutstopword_ratio_max等于common_word_withoutstopword_num除以问题1和问题2之间不含停用词的单词数量的最大值。
Then we have extracted fuzz_ratio, fuzz_partial_ratio, fuzz_token_set_ratio and fuzz_token_sort_ratio features with fuzzywuzzy string matching tool. Reference: https://github.com/seatgeek/fuzzywuzzy
然后我们使用fuzzywuzzy字符串匹配工具提取了fuzz_ratio、fuzz_partial_ratio、fuzz_token_set_ratio和fuzz_token_sort_ratio特征。参考：https://github.com/seatgeek/fuzzywuzzy
EDA with Features
基于特征的EDA
If First word or Last word is the same then there is a high chance that the question pairs are duplicates.
如果第一个单词或最后一个单词相同，则问题对为重复的可能性很高。
The number of total unique words (q1 and q2 both combined) with and without stopwords is less if question pairs are duplicate.
对于重复的问题对，总唯一单词数（q1和q2合并）在包含和不包含停用词的情况下都更少。
For duplicate question pairs, the total unique words to total words ratio is generally smaller.
对于重复的问题对，总唯一单词与总单词的比率通常更小。
Duplicate question pairs tend to have more common words between both the questions. Hence extracted features related to common words are also showing differences in distributions.
重复的问题对在两个问题之间往往有更多共同单词。因此，与共同单词相关的提取特征也显示出分布差异。
The fuzz ratios tend to be generally higher for duplicate question pairs.
fuzzy比率对于重复问题对通常更高。
Featurization with SentenceBERT
使用SentenceBERT进行特征化
We need to convert the questions to some numeric form to apply machine learning models. There are various options from basic like Bag of Words to Universal Sentence Encoder.
我们需要将问题转换为某种数值形式以应用机器学习模型。从基础的词袋模型到通用句子编码器，有多种选项。
I tried InferSent sentence embeddings. But it returns 4096 dimension representation. And after applying it the train data became huge. So I discarded it. And I chose SentenceBERT for this problem.
我尝试了InferSent句子嵌入。但它返回4096维的表示。应用之后训练数据变得巨大，所以我放弃了它。我选择了SentenceBERT来解决这个问题。
SentenceBERT is a BERT based sentence embedding technique. We will use pre-trained SentenceBERT model paraphrase-mpnet-base-v2, which is recommended for best quality. The SentenceBERT produces an output of 768 dimensions. https://www.sbert.net/
SentenceBERT是一种基于BERT的句子嵌入技术。我们将使用预训练的SentenceBERT模型paraphrase-mpnet-base-v2，该模型被推荐用于最佳质量。SentenceBERT产生768维的输出。https://www.sbert.net/
We created two more features cosine_simlarity_bert and euclidean_distance_bert which measures similarity and distance between both pairs of questions with SentenceBert representation.
我们创建了另外两个特征：cosine_similarity_bert和euclidean_distance_bert，它们使用SentenceBERT表示来衡量两对问题之间的相似性和距离。
The total number of features till now is 25.
到目前为止，特征总数为25个。
EDA on new features related toSentenceBERT
基于SentenceBERT相关新特征的EDA
Cosine Similarity is larger for duplicate pairs.
重复对的余弦相似度更大。
80% of non-duplicate question pairs and only 20% of duplicate question pairs have cosine similarity of <= .815
80%的非重复问题对和只有20%的重复问题对的余弦相似度 <= 0.815。
Euclidean Distance is smaller for duplicate pairs.
重复对的欧氏距离更小。
20% of non-duplicate question pairs and approx 80% of duplicate question pairs have euclidean distance of <= 2.
20%的非重复问题对和约80%的重复问题对的欧氏距离 <= 2。
It is showing the Pareto Principle (80–20 rule).
这显示了帕累托原则（80-20法则）。
Data Pre-processing
数据预处理
We normalized (min-max scaling) the extracted features. We have not normalized the embeddings because it is not recommended.
我们对提取的特征进行了归一化（最小-最大缩放）。我们没有对嵌入进行归一化，因为不推荐这样做。
We have 1561 features (25 + 768 + 768).
我们有1561个功能(25+768+768)。
25 are extracted features.
其中25个是提取的特征。
768+768 for sentence embedding of question 1 and question 2.
768+768是问题1和问题2的句子嵌入。
Since the dataset was imbalanced. We did oversample by sampling from the minority class.
由于数据集是不平衡的，我们通过对少数类进行采样来进行过采样。
Now we have 510048 data points for training. 255024 from each class.
现在我们有了510048个训练数据点。每个类别255024个。
Note that I have not set aside any data for testing locally. Because our main goal is to get a good score on Kaggle.
注意，我没有留出任何数据进行本地测试。因为我们的主要目标是在Kaggle上获得好分数。
Training Models
模型训练
Support Vector Classifier
支持向量分类器
While training Halving Grid Search CV with param grid,
在训练Halving Grid Search CV时，使用参数网格：
We have used LinearSVC because it is recommended for large datasets. We have used the L2 penalty and the loss function is squared of hinge loss. Also, it is recommended to use primal formulation for large datasets. For some values of C it was not conversing so I increased max_iter to 3000.
我们使用了LinearSVC，因为推荐用于大规模数据集。我们使用了L2惩罚，损失函数是hinge损失的平方。同时，推荐对大规模数据集使用原始公式。对于某些C值，模型没有收敛，所以我将max_iter增加到3000。0。
For cross-validation in halving grid search cv, I have used 1 shuffle split with a 70:30 split. Also, the scoring for selection is accuracy.
对于halving grid search cv中的交叉验证，我使用了1个shuffle split，按70：30划分。选择的评分指标是准确率。
The halving grid search cv found C=100 to be the best param. And the best accuracy is 85.79%. So the best estimator looks like,
halving grid search cv发现C=100是最佳参数。最佳准确率为85.79%。所以最佳估计器看起来是这样的：
Now since we need to minimize log loss for the competition. We would want a good predicted probability. Calibrated Classifier can be used to get a good predicted probability.
由于我们需要在竞赛中最小化对数损失，我们需要好的预测概率。可以使用Calibrated Classifier来获得好的预测概率。
After calibration of the model for probabilities. I predicted probabilities of test data and submitted on Kaggle. The public leader board score for the Kaggle submission is 0.36980. It is very good considering that the model assumes linear separability.
在对模型进行概率校准后，我预测了测试数据的概率并提交到Kaggle。Kaggle提交的公共排行榜得分为0.36980。考虑到模型假设线性可分性，这是非常好的成绩。
Random Forest
随机森林
You know Quora itself usage Random Forest for this problem. Or at least they did when they first posted the competition on Kaggle in June 2017.
你知道Quora本身就用随机森林来解决这个问题。或者至少在2017年6月首次在Kaggle上发布这个竞赛时他们是这样做的。
Same as before we are using halving grid search cv with following param grid,
和之前一样，我们使用halving grid search cv，参数网格如下：
And the rest of the params are the default for the Random Forest Classifier.
其余参数使用随机森林分类器的默认值。
We have used the very similar halving grid search cv as before,
我们使用了与之前非常相似的halving grid search cv：
The halving grid search cv found {‘max_depth’: 150, ‘min_samples_split’: 5, ‘n_estimators’: 800} to be the best params. And the best accuracy is 90.53%. So the accuracy has increased by 5% as compared to SVM. The best estimator looks like,
halving grid search cv发现{'max_depth'： 150， 'min_samples_split'： 5， 'n_estimators'： 800}是最佳参数。最佳准确率为90.53%。与SVM相比，准确率提高了5%。最佳估计器看起来是这样的：
Now at this point, I should have used calibration but because it has already taken a lot of time I skipped it. I should have used Bayesian Optimisation technique ?.
此时我应该使用校准，但由于已经花费了大量时间，我跳过了这一步。我应该使用贝叶斯优化技术的。
The public leader board score for the Kaggle submission is 0.32372, which slightly better than SVC. I was expecting a little less logloss but remember we have not done calibration (due to time constraints). We will try to better with XGBoost — the holy grail of ml models for the Kaggle competition.
Kaggle提交的公共排行榜得分为0.32372，比SVC略好。我预期的对数损失比这低一点，但请记住我们没有进行校准（由于时间限制）。我们将尝试用XGBoost——Kaggle竞赛中机器学习模型的圣杯——来做得更好。
XGBoost
XGBoost
Due to time and system configuration constrained, I decided to use 200000 data points to estimate a few of the params.
由于时间和系统配置的限制，我决定使用200000个数据点来估计部分参数。
At first, I was using Optuna for hyperparameter tuning but it had some issues because of which it was not releasing memory after the trials. So the system was crash after few trials.
起初我使用Optuna进行超参数调优，但它存在一些问题，导致在试验结束后没有释放内存，所以系统在几次试验后就崩溃了。
Later on, I decided to use HyperOpt for the tuning.
后来我决定使用HyperOpt进行调优。。
With HyperOpt, I tuned only max_depth and learning_rate. It was not a fine-tune because I used only 5 trials. But it gave a rough idea.
使用HyperOpt，我只调优了max_depth和learning_rate。这不是精细调优，因为我只使用了5次试验。但它提供了一个大致的思路。
Finally, I choose the following params for training the model on whole data,
最后，我选择了以下参数在全部数据上训练模型：
The objective = “binary:logistic” because we are trying to get probabilities. I have used tree_method = “hist” for faster training. grow_policy = “lossguide” is inspired from LightGBM for better accuracy.
objective = "binary：logistic"，因为我们试图获得概率。我使用tree_method = "hist"以加速训练。grow_policy = "lossguide"受到LightGBM的启发，以获得更好的准确率。
The num_boost_round is set to 600 with early_stopping_rounds as 20.
num_boost_round设置为600，early_stopping_rounds为20。
The public leader board score for the Kaggle submission is 0.32105, which slightly better than the other models. I was expecting a better result than this. Which is possible with more fine-tuning the hyperparameters. XGBoost have tons of hyperparameters https://xgboost.readthedocs.io/en/latest/parameter.html
Kaggle提交的公共排行榜得分为0.32105，比其他模型略好。我预期的结果比这更好。通过更精细的超参数调优是可以实现的。XGBoost有大量的超参数 https://xgboost.readthedocs.io/en/latest/parameter.html
Another XGBoost
另一个XGBoost
I was not happy with the result of the XGBoost model so I decided to tune the parameters with gut feeling.
我对XGBoost模型的结果不满意，所以我决定凭感觉调参。
The first thing I did is that I got rid of oversampled data by removing the duplicate rows.
我做的第一件事是移除了过采样的数据，去除了重复行。
This time I added a few more parameters to generalize better,
这次我添加了一些额外的参数以更好地泛化：
Also, I decreased the number of boosting round to 500.
同时，我将 boosting round 减少到500。
? Voila! We have a winner. This submission resulted in public LB score of 0.28170.
我们赢了！这次提交的公共LB得分为0.28170。
This seems a very good result.
这看起来是一个非常棒的结果。
Final Thoughts
最终思考
I learned a lot from this case study. I took some shortcuts either because of system configuration constraints or some time constraints.
从这个案例研究中我学到了很多。由于系统配置限制或时间限制，我走了一些捷径。
I also experienced firsthand that machine learning is not all about model building but steps before that take more time. The hyperparameter tuning can be automated but things like feature extraction or deciding on what featurization to use need to be done manually.
我也亲身经历了机器学习不仅仅是模型构建，在此之前步骤需要更多时间。超参数调优可以自动化，但像特征提取或决定使用哪种特征化方法这样的事情需要手动完成。
I spent almost two weeks ? and half of that time I was waiting for some execution to complete. So I think it’s a good idea to use things like Amazon SageMaker if you have resource-intensive tasks.
我花了将近两周时间，其中一半时间在等待某些执行完成。所以我认为，如果你有资源密集型任务，使用像Amazon SageMaker这样的东西是个好主意。
In the future, we can try some deep learning-based models.
将来，我们可以尝试一些基于深度学习的模型。
References
参考文献
a. https://appliedroots.com/
a. https://appliedroots.com/
