This GitHub repository contains all of the code and data for our final NLP class project. Our initial project proposal is available as well.
The original dataset and scorer script are located in the fnc-1/ directory. All additional data files can be found in the data/ directory, including the following:
- Generated features files for train and test:
feats.train.all.csvandfeats.competition_test.all.csv - Binary file containing 300-dimensional GloVe embeddings:
GoogleNews-vectors-negative300.bin - Serialized LDA topic model and dictionary:
lda.model.pklandlda.dct.pkl - "The New York Times Newswire Service" portion of the English Gigaword corpus:
nyt.txt - Subdirectory for sentiment analysis:
sentiment/ - Serialized count and TF-IDF vectorizer models:
vectorizer.count.pklandvectorizer.tfidf.pkl
All Python and bash scripts are included in the scripts/ directory. To run, create a conda environment and install the requirements:
[FakeNewsChallenge]$ conda create -n fnc-nyu python=3.6
[FakeNewsChallenge]$ source activate fnc-nyu
(fnc-nyu) [FakeNewsChallenge]$ pip install -r requirements.txt
Once your environment is set up, you can run the following scripts:
feature_builder.py: feature engineering modulerun_1stage.py: train and run 1-stage classifierrun_2stage.py: train and run 2-stage classifierscorer.py: original evaluation script published by the FNC-1 organizerstrain_lda_model.py: train LDA topic model on the NYT portion of Gigawordtune_xgb_params_1stage.py: tune hyperparameters for 1-stage classifiertune_xgb_params_2stage.py: tune hyperparameters for 2-stage classifierutils.py: utilty functions (configuration file import and text preprocessing)
Note that most of the scripts require that you pass a configuration file as argument:
(fnc-nyu) [FakeNewsChallenge]$ python run_1stage.py competition_test.yml
Most importantly, competition_test.yml contains the list of features that will be included or generated on each run.
Alternatively, use sbatch and the bash scripts below to run extensive Python jobs on HPC:
run_1stage.shrun_2stage.shtrain_lda_model.shtune_xgb_params_1stage.shtune_xgb_params_2stage.sh
There are two output files containing the predictions on the competition test set:
predictions.1stage.csv: contains the 1-stage classifier predictionspredictions.2stage.csv: contains the 2-stage classifier predictions
The top score after feature selection and hyperparameter tuning for the 1-stage classifier is **9128.5, or 78.35%.
(fnc) [mt3685@c38-15 FakeNewsChallenge]$ python scripts/scorer.py fnc-1/competition_test_stances.csv predictions.1stage.csv
CONFUSION MATRIX:
-------------------------------------------------------------
| | agree | disagree | discuss | unrelated |
-------------------------------------------------------------
| agree | 144 | 4 | 1607 | 148 |
-------------------------------------------------------------
| disagree | 12 | 1 | 522 | 162 |
-------------------------------------------------------------
| discuss | 190 | 2 | 3874 | 398 |
-------------------------------------------------------------
| unrelated | 2 | 0 | 246 | 18101 |
-------------------------------------------------------------
ACCURACY: 0.870
MAX - the best possible score (100% accuracy)
NULL - score as if all predicted stances were unrelated
TEST - score based on the provided predictions
|| MAX || NULL || TEST ||
|| 11651.25 || 4587.25 || 9128.5 ||
The top score after feature selection and hyperparameter tuning for the 2-stage classifier is 9161.5, or 78.63%.
(fnc) [mt3685@c38-15 FakeNewsChallenge]$ python scripts/scorer.py fnc-1/competition_test_stances.csv predictions.2stage.csv
CONFUSION MATRIX:
-------------------------------------------------------------
| | agree | disagree | discuss | unrelated |
-------------------------------------------------------------
| agree | 27 | 0 | 1733 | 143 |
-------------------------------------------------------------
| disagree | 9 | 0 | 533 | 155 |
-------------------------------------------------------------
| discuss | 45 | 0 | 4060 | 359 |
-------------------------------------------------------------
| unrelated | 5 | 0 | 366 | 17978 |
-------------------------------------------------------------
ACCURACY: 0.868
MAX - the best possible score (100% accuracy)
NULL - score as if all predicted stances were unrelated
TEST - score based on the provided predictions
|| MAX || NULL || TEST ||
|| 11651.25 || 4587.25 || 9161.5 ||
The original dataset is highly imbalanced, with the majority of example pairs being "unrelated":
| agree | disagree | discuss | unrelated | total |
|---|---|---|---|---|
| 3678 | 840 | 8909 | 36545 | 49972 |
To improve recall on "disagree" examples, we experimented with oversampling the training data to introduce a bias towards "disagree" predictions. After resampling, the number of "disagree" samples in the training data increases from 840 to 3678 (the original number of "agree" samples). When using resampling, the recall for "disagree" improves, while the precision for "discuss" decreases significantly.
The overall accuracy of the 2-stage classifier with resampling is 9115.75, or 78.24%.
(fnc) [mt3685@c38-15 FakeNewsChallenge]$ python scripts/scorer.py fnc-1/competition_test_stances.csv predictions.2stage.csv
CONFUSION MATRIX:
-------------------------------------------------------------
| | agree | disagree | discuss | unrelated |
-------------------------------------------------------------
| agree | 25 | 21 | 1718 | 139 |
-------------------------------------------------------------
| disagree | 4 | 7 | 529 | 157 |
-------------------------------------------------------------
| discuss | 33 | 84 | 3993 | 354 |
-------------------------------------------------------------
| unrelated | 6 | 3 | 366 | 17974 |
-------------------------------------------------------------
ACCURACY: 0.866
MAX - the best possible score (100% accuracy)
NULL - score as if all predicted stances were unrelated
TEST - score based on the provided predictions
|| MAX || NULL || TEST ||
|| 11651.25 || 4587.25 || 9115.75 ||