Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Kaggle for downloading household data #61

Open
wants to merge 15 commits into
base: feature/correlation-similarity
Choose a base branch
from
Open

Use Kaggle for downloading household data #61

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
18d9f80
Modify makefile to use Kaggle for downloading household data
gmingas Apr 12, 2021
c8b8ea6
Add kaggle username and key to CI script
gmingas Apr 12, 2021
2f3828f
Fix name of kaggle username variable
gmingas Apr 12, 2021
bd6d60b
Remove erroneous lines from makefile
gmingas Apr 12, 2021
eb1cae6
Remove erroneous lines in makefile
OscartGiles Apr 12, 2021
148ad6b
Merge branch 'feature/kaggle-automation' of https://github.com/alan-t…
OscartGiles Apr 12, 2021
8473f69
Add creation of Kaggle token file from environment variables
gmingas Apr 12, 2021
3cd5ac6
Try to read secret
OscartGiles Apr 12, 2021
6d8b25b
Merge branch 'feature/kaggle-automation' of https://github.com/alan-t…
gmingas Apr 12, 2021
c2bf876
Remove echo
OscartGiles Apr 12, 2021
1220aa3
Merge branch 'feature/kaggle-automation' of https://github.com/alan-t…
OscartGiles Apr 12, 2021
923c1f8
Add directory creation for kaggle in makefile
gmingas Apr 12, 2021
c588013
Merge branch 'feature/kaggle-automation' of https://github.com/alan-t…
gmingas Apr 12, 2021
76bf9fb
Use environment vars
OscartGiles Apr 12, 2021
a14f834
Merge branch 'feature/kaggle-automation' of https://github.com/alan-t…
OscartGiles Apr 12, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/workflows/run-synth-pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,4 +21,7 @@ jobs:
run: docker build -f dockerfiles/quipp-dev.Dockerfile -t turinginst/quipp-env:latest .
-
name: Run pipeline
run: docker run turinginst/quipp-env:latest make
env:
KAGGLE_USERNAME: ${{ secrets.KAGGLE_USERNAME }}
KAGGLE_KEY: ${{ secrets.KAGGLE_KEY }}
run: docker run -e KAGGLE_USERNAME=$KAGGLE_USERNAME -e KAGGLE_KEY=$KAGGLE_KEY turinginst/quipp-env:latest make
14 changes: 10 additions & 4 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ ADD_PROVENANCE = $(PROVENANCE_DEF) && provenance
## set data file paths
AE_DEIDENTIFIED_DATA = generator-outputs/odi-nhs-ae/hospital_ae_data_deidentify.csv generator-outputs/odi-nhs-ae/hospital_ae_data_deidentify.json
LONDON_POSTCODES = generators/odi-nhs-ae/data/London\ postcodes.csv
HH_DATA = generators/household_poverty/data\ train.csv
HP_DATA_CLEAN = generator-outputs/household_poverty/train_cleaned.csv generator-outputs/household_poverty/train_cleaned.json
generated-data: $(AE_DEIDENTIFIED_DATA) $(HP_DATA_CLEAN)

Expand All @@ -62,6 +63,13 @@ $(LONDON_POSTCODES):
curl -o "./data/London postcodes.csv" \
https://www.doogal.co.uk/UKPostcodesCSV.ashx?region=E12000007

# download the Household poverty dataset from Kaggle using API
$(HH_DATA):
cd generators/household_poverty/data/ && \
kaggle competitions download -c costa-rican-household-poverty-prediction && \
unzip costa-rican-household-poverty-prediction.zip -d unzipped && \
cp unzipped/train.csv .

# make the "A&E deidentified" generated dataset
# this is currently the only generated dataset, so it is handled with
# its own rule
Expand All @@ -70,12 +78,10 @@ $(AE_DEIDENTIFIED_DATA) &: $(LONDON_POSTCODES)
mkdir -p generator-outputs/household_poverty/ && \
cd generator-outputs/odi-nhs-ae/ && \
$(PYTHON) $(QUIPP_ROOT)/generators/odi-nhs-ae/generate.py && \
$(PYTHON) $(QUIPP_ROOT)/generators/odi-nhs-ae/deidentify.py && \
cd ../household_poverty/ && \
$(PYTHON) $(QUIPP_ROOT)/generators/household_poverty/clean.py
$(PYTHON) $(QUIPP_ROOT)/generators/odi-nhs-ae/deidentify.py

# pre-process the Household Poverty dataset
$(HP_DATA_CLEAN):
$(HP_DATA_CLEAN): $(HH_DATA)
mkdir -p generator-outputs/household_poverty/ && \
cd generator-outputs/household_poverty/ && \
$(PYTHON) $(QUIPP_ROOT)/generators/household_poverty/clean.py
Expand Down
1 change: 1 addition & 0 deletions env-configuration/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,4 @@ shap
ipython
numpy>=1.20
pulp
kaggle