Sk ez pp/hist data load #2730

gsa-suk · 2023-11-03T22:33:12Z

#2718

census_to_gsafac application containing two management commands

fac_s3 command to load census csv files from a local data folder to fac-census-to-gsafac-s3 bucket.
csv_to_postgres command to load data from csv files in fac-census-to-gsafac-s3/data bucket into postgres database.

README file contains details about how to run the commands.

Upon completion of the commands, the following messages will be displayed in the console:

PR checklist: submitters

PR checklist: reviewers

Pull the branch to your local environment and run make docker-clean; make docker-first-run && docker compose up; then run docker compose exec web /bin/bash -c "python manage.py test"
Manually test out the changes locally, or check this box to verify that it wasn’t applicable in this case.
Check that the PR has appropriate tests. Look out for changes in HTML/JS/JSON Schema logic that may need to be captured in Python tests even though the logic isn’t in Python.
Verify that no Git surgery is necessary at any point (such as during a merge party), or, if it was, repeat the testing after it’s finished.

The larger the PR, the stricter we should be about these points.

github-actions · 2023-11-03T22:33:58Z

Terraform plan for meta

No changes. Your infrastructure matches the configuration.

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

Warning: Argument is deprecated

  with module.s3-backups.cloudfoundry_service_instance.bucket,
  on /tmp/terraform-data-dir/modules/s3-backups/s3/main.tf line 14, in resource "cloudfoundry_service_instance" "bucket":
  14:   recursive_delete = var.recursive_delete

Since CF API v3, recursive delete is always done on the cloudcontroller side.
This will be removed in future releases

✅ Plan applied in Deploy to Development and Management Environment #350

github-actions · 2023-11-03T22:34:00Z

Terraform plan for dev

No changes. Your infrastructure matches the configuration.

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

Warning: Argument is deprecated

  with module.dev.module.database.cloudfoundry_service_instance.rds,
  on /tmp/terraform-data-dir/modules/dev.database/database/main.tf line 14, in resource "cloudfoundry_service_instance" "rds":
  14:   recursive_delete = var.recursive_delete

Since CF API v3, recursive delete is always done on the cloudcontroller side.
This will be removed in future releases

(and 2 more similar warnings elsewhere)

✅ Plan applied in Deploy to Development and Management Environment #350

github-actions · 2023-11-03T22:56:12Z

File	Coverage		Missing
All files	`86%`	✅
api/serializers.py	`88%`	✅	`177-178` `183` `188`
api/test_views.py	`95%`	✅	`103`
api/uei.py	`88%`	✅	`87` `118-119` `163` `167-168`
api/views.py	`98%`	✅	`195-196` `334-335`
audit/file_downloads.py	`73%`	❌	`35-53` `81-83`
audit/forms.py	`47%`	❌	`22-29` `142-149`
audit/intake_to_dissemination.py	`92%`	✅	`67-68` `201-207` `257`
audit/models.py	`84%`	❌	`58` `60` `65` `67` `214` `226-229` `247` `420` `438-439` `447` `469` `558-559` `563` `571` `580` `586`
audit/test_commands.py	`87%`	✅
audit/test_mixins.py	`90%`	✅	`112-113` `117-119` `184-185` `189-191`
audit/test_validators.py	`95%`	✅	`436` `440` `608-609` `848` `855` `862` `869`
audit/test_views.py	`95%`	✅	`410-442` `451-482` `491-519`
audit/test_workbooks_should_fail.py	`88%`	✅	`56` `83-84` `88`
audit/test_workbooks_should_pass.py	`90%`	✅	`56` `81`
audit/utils.py	`70%`	❌	`13` `21` `33-35` `38`
audit/validators.py	`92%`	✅	`137` `189` `283-292` `299-308` `486-490` `495-499` `515-524`
audit/views.py	`31%`	❌	`90-111` `134-135` `209-210` `255-256` `267-268` `270-274` `321-334` `337-351` `356-369` `386-392` `397-417` `420-448` `453-482` `485-529` `534-554` `557-585` `590-619` `622-666` `671-683` `686-696` `701-713` `740-741` `746-795` `798-838` `841-858`
audit/cross_validation/additional_ueis.py	`93%`	✅	`33`
audit/cross_validation/check_award_ref_declaration.py	`90%`	✅
audit/cross_validation/check_award_reference_uniqueness.py	`93%`	✅
audit/cross_validation/check_certifying_contacts.py	`87%`	✅
audit/cross_validation/check_findings_count_consistency.py	`91%`	✅
audit/cross_validation/check_ref_number_in_cap.py	`90%`	✅
audit/cross_validation/check_ref_number_in_findings_text.py	`90%`	✅
audit/cross_validation/errors.py	`78%`	❌	`30` `69`
audit/cross_validation/naming.py	`93%`	✅	`182`
audit/cross_validation/submission_progress_check.py	`95%`	✅	`79`
audit/cross_validation/tribal_data_sharing_consent.py	`81%`	❌	`33` `36` `40`
audit/cross_validation/validate_general_information.py	`93%`	✅	`28-29`
audit/fixtures/single_audit_checklist.py	`55%`	❌	`146-183` `229-238`
audit/intakelib/exceptions.py	`71%`	❌	`7-9` `12`
audit/intakelib/intermediate_representation.py	`91%`	✅	`27-28` `73` `91` `129` `162` `200-203` `212-213`
audit/intakelib/mapping_audit_findings.py	`97%`	✅	`53`
audit/intakelib/mapping_audit_findings_text.py	`97%`	✅	`52`
audit/intakelib/mapping_federal_awards.py	`93%`	✅	`95`
audit/intakelib/mapping_util.py	`81%`	❌	`21` `25` `29` `99` `104-105` `114-120` `130` `145` `150`
audit/intakelib/checks/check_all_unique_award_numbers.py	`79%`	❌	`24`
audit/intakelib/checks/check_cardinality_of_passthrough_names_and_ids.py	`91%`	✅
audit/intakelib/checks/check_cluster_total.py	`85%`	✅	`49` `65`
audit/intakelib/checks/check_finding_prior_references_pattern.py	`73%`	❌	`33` `43-44`
audit/intakelib/checks/check_findings_grid_validation.py	`84%`	❌	`58`
audit/intakelib/checks/check_has_all_the_named_ranges.py	`84%`	❌	`52`
audit/intakelib/checks/check_is_a_workbook.py	`69%`	❌	`20`
audit/intakelib/checks/check_loan_balance_entries.py	`78%`	❌	`22` `39-40`
audit/intakelib/checks/check_loan_balance_present.py	`76%`	❌	`27` `36`
audit/intakelib/checks/check_look_for_empty_rows.py	`91%`	✅	`18`
audit/intakelib/checks/check_no_major_program_no_type.py	`76%`	❌	`18` `27`
audit/intakelib/checks/check_no_repeat_findings.py	`76%`	❌	`21` `30`
audit/intakelib/checks/check_other_cluster_names.py	`81%`	❌	`24` `34`
audit/intakelib/checks/check_passthrough_name_when_no_direct.py	`88%`	✅	`9` `47`
audit/intakelib/checks/check_sequential_award_numbers.py	`76%`	❌	`14` `22`
audit/intakelib/checks/check_show_ir.py	`70%`	❌	`8-14`
audit/intakelib/checks/check_start_and_end_rows_of_all_columns_are_same.py	`89%`	✅	`14`
audit/intakelib/checks/check_state_cluster_names.py	`65%`	❌	`23-24` `34`
audit/intakelib/checks/check_version_number.py	`73%`	❌	`21` `31-32`
audit/intakelib/checks/runners.py	`95%`	✅	`129`
audit/intakelib/common/util.py	`89%`	✅	`21` `38`
audit/intakelib/transforms/xform_reformat_prior_references.py	`55%`	❌	`12-17`
audit/intakelib/transforms/xform_rename_additional_notes_sheet.py	`81%`	❌	`14`
audit/management/commands/load_fixtures.py	`46%`	❌	`39-45`
audit/viewlib/submission_progress_view.py	`89%`	✅	`111` `171-172`
audit/viewlib/tribal_data_consent.py	`34%`	❌	`23-41` `44-79`
audit/viewlib/unlock_after_certification.py	`57%`	❌	`28-47` `69-83`
audit/viewlib/upload_report_view.py	`26%`	❌	`32-35` `44` `91-117` `120-170` `178-209`
config/urls.py	`71%`	❌	`83`
dissemination/models.py	`99%`	✅	`460`
dissemination/search.py	`51%`	❌	`19` `69-97` `104-114`
dissemination/views.py	`62%`	❌	`29-31` `34-92` `159` `161` `163`
dissemination/migrations/0002_general_fac_accepted_date.py	`47%`	❌	`10-12`
djangooidc/backends.py	`78%`	❌	`32` `57-63`
djangooidc/exceptions.py	`66%`	❌	`19` `21` `23` `28`
djangooidc/oidc.py	`16%`	❌	`32-35` `45-51` `64-70` `92-149` `153-199` `203-226` `230-275` `280-281` `286`
djangooidc/views.py	`80%`	❌	`22` `43` `114`
djangooidc/tests/common.py	`96%`	✅
report_submission/forms.py	`92%`	✅	`35`
report_submission/views.py	`76%`	❌	`83` `215-216` `218` `240-241` `260-261` `287-396` `399-409`
report_submission/templatetags/get_attr.py	`76%`	❌	`8` `11-14` `18`
support/admin.py	`88%`	✅	`76` `79` `84` `91-97` `100-102`
support/cog_over.py	`91%`	✅	`30-33` `93` `145`
support/models.py	`89%`	✅	`103-104`
support/test_cog_over.py	`98%`	✅	`134-135` `224`
support/management/commands/seed_cog_baseline.py	`98%`	✅	`20-21`
tools/update_program_data.py	`89%`	✅	`96`
users/auth.py	`95%`	✅	`40-41`
users/models.py	`97%`	✅	`51-52`
users/fixtures/user_fixtures.py	`91%`	✅

Minimum allowed coverage is 85%

Generated by 🐒 cobertura-action against 7396ca8

Co-authored-by: Purvin Patel <[email protected]>

…_load

* Initial commit * Initial commit * Initial commit * Initial commit * Initial commit * Initial commit * Initial commit * Initial commit * Removed comments * Added census_to_gsafac * Initial commit * Initial commit * Initial commit * Initial commit * Initial commit * Added procedure to load test Census data to postgres * Excluding workbook loader * Excluding workbook loader * Excluding load_raw * Updates * Added c2g-db * Added c2g-db * Replaced c2g with census_to_gsafac, renamed raw_to_pg.py as csv_to_postgres.py * Replaced c2g with census_to_gsafac, renamed raw_to_pg.py as csv_to_postgres.py * Replaced c2g with census_to_gsafac, renamed raw_to_pg.py as csv_to_postgres.py * Replaced c2g with census_to_gsafac, renamed raw_to_pg.py as csv_to_postgres.py * Replaced c2g with census_to_gsafac, renamed raw_to_pg.py as csv_to_postgres.py * Replaced c2g with census_to_gsafac, renamed raw_to_pg.py as csv_to_postgres.py * Replaced c2g with census_to_gsafac, renamed raw_to_pg.py as csv_to_postgres.py * Apply suggestions from code review Co-authored-by: Hassan D. M. Sambo <[email protected]> * Added census-to-gsafac database * Replaced c2g with census-to-gsafac * Fix linting * Fix linting * Fix linting * Fix linting * Fix linting * Reformatted with black * Reformatted with black * Reformatted with black * Updated S3 bucket name and filename * Updated S3 bucket name and filename * Updates * Consolidated census_to_gsafac and census_historical_migration apps * Django migration * Telling mypy to ignore django migration files * Linting * Incorporated chunking capabilities from Alternative suggestion for loading data from S3 #2660 * Incorporated chunking capabilities from Alternative suggestion for loading data from S3 #2660 * Moving fac_s3.py to support/management/commands/ * Moving fac_s3.py to support/management/commands/ * Added load_data function * Tested load_data * Removed import botocore * Removed import botocore * refactored csv_to_postgres.py Co-authored-by: Purvin Patel <[email protected]> * added chunk-size arguments * added help comments for load_data * Code cleaning * Renamed chunk-size to chunksize * Added chunksize argument --------- Co-authored-by: SudhaUKumar <[email protected]> Co-authored-by: Purvin Patel <[email protected]> Co-authored-by: Hassan D. M. Sambo <[email protected]> Co-authored-by: Edward Zapata <[email protected]> Co-authored-by: Purvin Patel <[email protected]>

SudhaUKumar added 22 commits November 2, 2023 15:08

Initial commit

8e387fb

Initial commit

cd7dc1e

Initial commit

cd4b87b

Initial commit

1adc362

Initial commit

54ba1df

Initial commit

ff307ec

Initial commit

b8498e5

Initial commit

1138a7c

Removed comments

3c77f09

Added census_to_gsafac

2547ef4

Initial commit

9173ba6

Initial commit

352456e

Initial commit

988a111

Initial commit

71ad2cc

Initial commit

d6438e6

Added procedure to load test Census data to postgres

ae5ebc1

Excluding workbook loader

356a3d3

Excluding workbook loader

47d034f

Excluding load_raw

4f4f90c

Updates

8db3fe2

Added c2g-db

2efada7

Added c2g-db

9532f1e

gsa-suk marked this pull request as draft November 3, 2023 22:33

gsa-suk temporarily deployed to dev November 3, 2023 22:33 — with GitHub Actions Inactive

gsa-suk temporarily deployed to meta November 3, 2023 22:33 — with GitHub Actions Inactive

gsa-suk requested a review from danswick November 3, 2023 22:33

gsa-suk requested a review from jadudm November 3, 2023 22:37

gsa-suk temporarily deployed to meta November 8, 2023 17:24 — with GitHub Actions Inactive

gsa-suk marked this pull request as ready for review November 8, 2023 18:15

JeanMarie-PM approved these changes Nov 8, 2023

View reviewed changes

refactored csv_to_postgres.py

f951ea2

Co-authored-by: Purvin Patel <[email protected]>

eazapata23 temporarily deployed to dev November 8, 2023 21:06 — with GitHub Actions Inactive

eazapata23 temporarily deployed to meta November 8, 2023 21:06 — with GitHub Actions Inactive

Edward Zapata added 2 commits November 8, 2023 16:58

added chunk-size arguments

4f03d7a

added help comments for load_data

13736ad

eazapata23 temporarily deployed to dev November 8, 2023 22:10 — with GitHub Actions Inactive

eazapata23 temporarily deployed to meta November 8, 2023 22:10 — with GitHub Actions Inactive

Code cleaning

e3f90a1

sambodeme temporarily deployed to dev November 8, 2023 22:47 — with GitHub Actions Inactive

sambodeme temporarily deployed to meta November 8, 2023 22:47 — with GitHub Actions Inactive

Renamed chunk-size to chunksize

653ec97

gsa-suk temporarily deployed to dev November 9, 2023 00:15 — with GitHub Actions Inactive

gsa-suk temporarily deployed to meta November 9, 2023 00:15 — with GitHub Actions Inactive

Added chunksize argument

7245368

gsa-suk temporarily deployed to dev November 9, 2023 00:21 — with GitHub Actions Inactive

gsa-suk temporarily deployed to meta November 9, 2023 00:21 — with GitHub Actions Inactive

Merge branch 'main' of github.com:GSA-TTS/FAC into sk_ez_pp/hist_data…

7396ca8

…_load

gsa-suk temporarily deployed to dev November 9, 2023 00:25 — with GitHub Actions Inactive

gsa-suk temporarily deployed to meta November 9, 2023 00:25 — with GitHub Actions Inactive

gsa-suk enabled auto-merge November 9, 2023 00:31

sambodeme approved these changes Nov 9, 2023

View reviewed changes

gsa-suk added this pull request to the merge queue Nov 9, 2023

Merged via the queue into main with commit 59f35f5 Nov 9, 2023
13 checks passed

gsa-suk deleted the sk_ez_pp/hist_data_load branch November 9, 2023 00:49

phildominguez-gsa added the census label Nov 9, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sk ez pp/hist data load #2730

Sk ez pp/hist data load #2730

gsa-suk commented Nov 3, 2023 •

edited

Loading

github-actions bot commented Nov 3, 2023 •

edited

Loading

github-actions bot commented Nov 3, 2023 •

edited

Loading

github-actions bot commented Nov 3, 2023 •

edited

Loading

Sk ez pp/hist data load #2730

Sk ez pp/hist data load #2730

Conversation

gsa-suk commented Nov 3, 2023 • edited Loading

PR checklist: submitters

PR checklist: reviewers

github-actions bot commented Nov 3, 2023 • edited Loading

github-actions bot commented Nov 3, 2023 • edited Loading

github-actions bot commented Nov 3, 2023 • edited Loading

gsa-suk commented Nov 3, 2023 •

edited

Loading

github-actions bot commented Nov 3, 2023 •

edited

Loading

github-actions bot commented Nov 3, 2023 •

edited

Loading

github-actions bot commented Nov 3, 2023 •

edited

Loading