Merge pull request #5 from iag-geo/202211

Updates for 202211 Geoscape Release
iag-geo · Nov 23, 2022 · d961939 · d961939
2 parents 6e165ce + 2616dae
commit d961939
Show file tree

Hide file tree

Showing 17 changed files with 105 additions and 349 deletions.
diff --git a/README.md b/README.md
@@ -1,14 +1,14 @@
 # Concord
 
-A [CSV file](https://minus34.com/opendata/geoscape-202208/boundary_concordance.csv) and supporting scripts for converting data between Australian boundaries.
+A [CSV file](https://minus34.com/opendata/geoscape-202211/boundary_concordance.csv) and supporting scripts for converting data between Australian boundaries.
 
 It solves the problem of trying to merge 2 or more datasets based on different census or administrative boundaries such as statistical areas or postcodes.
 
 It does this by providing a list of **_concordances_** between pairs of boundaries.  _e.g. In the image below: 100% of postcode 3126 fits within the Boroondara LGA. However, only ~46% of postcode 3127 fits within that LGA._
 
 In this context, **_concordance_** describes what % of residential addresses in a "from" boundary fit within a "to" boundary.
 
-Download & import the ~50Mb [concordance file](https://minus34.com/opendata/geoscape-202208/boundary_concordance.csv) into your database or reporting tool to [get started](#get-started). A [script](/postgres-scripts/00_import_concordance_file.sql) for importing into Postgres is also provided.
+Download & import the ~50Mb [concordance file](https://minus34.com/opendata/geoscape-202211/boundary_concordance.csv) into your database or reporting tool to [get started](#get-started). A [script](/postgres-scripts/00_import_concordance_file.sql) for importing into Postgres is also provided.
 
 ### Example Use Cases
 
@@ -42,7 +42,7 @@ In the [score](/data/boundary_concordance_score.csv) file, the **_error_** measu
 
 The concordance file is generated by the following process:
 
-1. Tag all GNAF addresses with 2016 & 2021 ABS Census boundaries and geoscape 202208 Administrative boundaries
+1. Tag all GNAF addresses with 2016 & 2021 ABS Census boundaries and geoscape 202211 Administrative boundaries
 2. Remove all addresses in non-residential ABS Census 2021 meshblocks
 3. Aggregate all residential addresses by a set of _**from**_ boundary and _**to**_ boundary pairs (e.g. postcode to LGA)
 4. Determine the % overlap of residential addresses between both boundary types for all boundary pairs
@@ -63,7 +63,7 @@ There are 2 options to get the data:
 
 #### 1. Download and Import
 
-1. Download the [concordance file](https://minus34.com/opendata/geoscape-202208/boundary_concordance.csv)
+1. Download the [concordance file](https://minus34.com/opendata/geoscape-202211/boundary_concordance.csv)
 2. Import it into your database/reporting tool of choice. If using Postgres:
     1. Edit the file path, schema name & table owner in `00_import_concordance_file.sql` in the [postgres-scripts](/postgres-scripts) folder
     2. Run the SQL script to import the file  
@@ -75,9 +75,9 @@ This requires a knowledge of Python, Postgres & pg_restore.
 BTW - if the boundary combination you want isn't in the default concordance file - you need to edit the `settings.py` file before running `create_concordance_file.py`. If this is too hard - raise an [issue](https://github.com/iag-geo/concord/issues) and we may be able to generate it for you; noting you shouldn't convert data to a smaller boundary due to the increase in data errors.
 
 **Running the script only needs to be done for 3 reasons:**
-1. The boundary from/to combination you need isn't in the standard [concordances file](https://minus34.com/opendata/geoscape-202208/boundary_concordance.csv)
+1. The boundary from/to combination you need isn't in the standard [concordances file](https://minus34.com/opendata/geoscape-202211/boundary_concordance.csv)
 2. It's now the future and we've been too lazy to update the concordances file with the latest boundary data from the ABS and/or Geoscape
-3. You have a license of [Geoscape Buildings](https://geoscape.com.auhttps://minus34.com/opendata/geoscape-202208/boundary_concordance.csv/buildings/) or [Geoscape Land Parcels](https://geoscape.com.auhttps://minus34.com/opendata/geoscape-202208/boundary_concordance.csv/land-parcels/) and want to use the _planning zone_ data in those products to:
+3. You have a license of [Geoscape Buildings](https://geoscape.com.auhttps://minus34.com/opendata/geoscape-202211/boundary_concordance.csv/buildings/) or [Geoscape Land Parcels](https://geoscape.com.auhttps://minus34.com/opendata/geoscape-202211/boundary_concordance.csv/land-parcels/) and want to use the _planning zone_ data in those products to:
     1. Use a more accurate list of residential addresses to determine the data apportionment percentages (see **note** below); or
     2. Use a different set of addresses to apportion your data; e.g. industrial or commercial addresses
 
@@ -88,8 +88,8 @@ BTW - if the boundary combination you want isn't in the default concordance file
 Running the script requires the following open data, available as Postgres dump files, as well as the optional licensed Geoscape data mentioned above:
 1. ABS Census 2016 boundaries ([download](https://minus34.com/opendata/census-2016/census_2016_bdys.dmp))
 2. ABS Census 2021 boundaries ([download](https://minus34.com/opendata/census-2021/census_2021_bdys_gda94.dmp))
-3. GNAF from gnaf-loader ([download](https://minus34.com/opendata/geoscape-202208/gnaf-202208.dmp))
-4. Geoscape Administrative Boundaries from gnaf-loader ([download](https://minus34.com/opendata/geoscape-202208/admin-bdys-202208.dmp))
+3. GNAF from gnaf-loader ([download](https://minus34.com/opendata/geoscape-202211/gnaf-202211.dmp))
+4. Geoscape Administrative Boundaries from gnaf-loader ([download](https://minus34.com/opendata/geoscape-202211/admin-bdys-202211.dmp))
 5. ABS Census 2016 data - used to generate error rates only ([download](https://minus34.com/opendata/census-2016/census_2016_data.dmp))
 
 #### Process
@@ -100,7 +100,7 @@ Running the script requires the following open data, available as Postgres dump
     2. Edit the `02_create_residential_address_table.sql` in the [postgres-scripts](/postgres-scripts/data-prep) folder to suit your dataset and schema name
     3. Run the above SQL script
 3. Review & edit the boundaries to process in `settings.py` as required - make any required changes in the sections near the bottom marked for editing. If optionally using Geoscape Buildings data for residential address - change the `residential_address_source` to use it.
-4. Add `psycopg2` to your Python 3.x environment
+4. Add `psycopg` to your Python 3.x environment
 5. Run the script (takes ~30-45 mins on a 2017 MacBook Pro with 16Gb of RAM and 4 cores)
 
 #### Command Line Options
@@ -117,7 +117,7 @@ The behaviour of the Python script can be controlled by specifying various comma
 * `--pgpassword` password for accessing the Postgres server. This defaults to the `PGPASSWORD` environment variable if set, otherwise `password`.
 
 ##### Optional Arguments
-* `--geoscape-version` Geoscape version number in YYYYMM format. Defaults to current year and last release month. e.g. `202208`.
+* `--geoscape-version` Geoscape version number in YYYYMM format. Defaults to current year and last release month. e.g. `202211`.
 * `--gnaf-schema` input schema name to store final GNAF tables in. Also the **output schema** for the concordance table. Defaults to `gnaf_<geoscape_version>`.
 * `--admin-schema` input schema name to store final admin boundary tables in. Defaults to `admin_bdys_<geoscape_version>`.
 * `--output-table` name of both output concordance table and file. Defaults to `boundary_concordance`.
@@ -147,8 +147,8 @@ WITH pc_data AS (
           con.to_name AS lga_name,
           sum(pc.cases::float * con.address_percent / 100.0)::integer AS cases
    FROM testing.nsw_covid_cases_20220503_postcode AS pc
-           INNER JOIN gnaf_202208.boundary_concordance AS con ON pc.postcode = con.from_id
-   WHERE con.from_source = 'geoscape 202208'
+           INNER JOIN gnaf_202211.boundary_concordance AS con ON pc.postcode = con.from_id
+   WHERE con.from_source = 'geoscape 202211'
      AND con.from_bdy = 'postcode'
      AND con.to_source = 'abs 2016'
      AND con.to_bdy = 'lga'
@@ -167,9 +167,9 @@ FROM testing.nsw_covid_tests_20220503_lga AS lga
 
 ## Data Licenses
 
-Incorporates or developed using G-NAF © [Geoscape Australia](https://geoscape.com.au/legalhttps://minus34.com/opendata/geoscape-202208/boundary_concordance.csv-copyright-and-disclaimer/) licensed by the Commonwealth of Australia under the [Open Geo-coded National Address File (G-NAF) End User Licence Agreement](https:/https://minus34.com/opendata/geoscape-202208/boundary_concordance.csv.gov.auhttps://minus34.com/opendata/geoscape-202208/boundary_concordance.csvset/ds-dga-19432f89-dc3a-4ef3-b943-5326ef1dbecc/distribution/dist-dga-09f74802-08b1-4214-a6ea-3591b2753d30/details?q=).
+Incorporates or developed using G-NAF © [Geoscape Australia](https://geoscape.com.au/legalhttps://minus34.com/opendata/geoscape-202211/boundary_concordance.csv-copyright-and-disclaimer/) licensed by the Commonwealth of Australia under the [Open Geo-coded National Address File (G-NAF) End User Licence Agreement](https:/https://minus34.com/opendata/geoscape-202211/boundary_concordance.csv.gov.auhttps://minus34.com/opendata/geoscape-202211/boundary_concordance.csvset/ds-dga-19432f89-dc3a-4ef3-b943-5326ef1dbecc/distribution/dist-dga-09f74802-08b1-4214-a6ea-3591b2753d30/details?q=).
 
-Incorporates or developed using Administrative Boundaries © [Geoscape Australia](https://geoscape.com.au/legalhttps://minus34.com/opendata/geoscape-202208/boundary_concordance.csv-copyright-and-disclaimer/) licensed by the Commonwealth of Australia under [Creative Commons Attribution 4.0 International licence (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/).
+Incorporates or developed using Administrative Boundaries © [Geoscape Australia](https://geoscape.com.au/legalhttps://minus34.com/opendata/geoscape-202211/boundary_concordance.csv-copyright-and-disclaimer/) licensed by the Commonwealth of Australia under [Creative Commons Attribution 4.0 International licence (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/).
 
 Based on [Australian Bureau of Statistics](https://www.abs.gov.au/websitedbs/d3310114.nsf/Home/Attributing+ABS+Material) data, licensed by the Commonwealth of Australia under [Creative Commons Attribution 4.0 International licence (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/).
 

diff --git a/create_concordance_file.py b/create_concordance_file.py
@@ -2,15 +2,15 @@
 import geoscape
 import logging
 import os
-import psycopg2  # need to install package
+import psycopg  # need to install package
 import settings  # gets global vars and runtime arguments
 
 from datetime import datetime
 
 
 def main():
     # connect to Postgres database
-    pg_conn = psycopg2.connect(settings.pg_connect_string)
+    pg_conn = psycopg.connect(settings.pg_connect_string)
     pg_conn.autocommit = True
     pg_cur = pg_conn.cursor()
 
@@ -192,7 +192,7 @@ def add_asgs_concordances(pg_cur):
                                    count(*) as address_count,
                                    100.0 as address_percent
                             from census_2016_bdys.mb_2016_aust as mb
-                            inner join gnaf_202208.address_principals as gnaf on gnaf.mb_2016_code::text = mb.mb_code16
+                            inner join gnaf_202211.address_principals as gnaf on gnaf.mb_2016_code::text = mb.mb_code16
                             group by from_id,
                                      from_name,
                                      to_id,
@@ -240,7 +240,7 @@ def add_asgs_concordances(pg_cur):
                                    count(*) as address_count,
                                    100.0 as address_percent
                             from census_2021_bdys_gda94.mb_2021_aust_gda94 as mb
-                                     inner join gnaf_202208.address_principals as gnaf 
+                                     inner join gnaf_202211.address_principals as gnaf 
                                          on gnaf.mb_2021_code::text = mb.mb_code_2021
                             group by from_id,
                                      from_name,
@@ -449,8 +449,11 @@ def export_to_csv(pg_cur, table, file_name):
                              to_source, 
                              to_bdy
                 ) TO STDOUT WITH CSV HEADER"""
-    with open(os.path.join(settings.output_path, file_name), "w") as f:
-        pg_cur.copy_expert(query, f)
+
+    with open(os.path.join(settings.output_path, file_name), "wb") as f:
+        with pg_cur.copy(query) as copy:
+            while data := copy.read():
+                f.write(data)
 
 
 if __name__ == "__main__":

diff --git a/data/boundary_concordance_score.csv b/data/boundary_concordance_score.csv
@@ -1,9 +1,9 @@
 from_source,from_bdy,to_source,to_bdy,concordance_percent,error_percent
 abs 2016,gcc,abs 2021,gccsa,100,
-abs 2016,lga,abs 2016,sa3,74,4.7
+abs 2016,lga,abs 2016,sa3,73,4.7
 abs 2016,lga,abs 2016,ste,100,0.2
 abs 2016,poa,abs 2016,lga,93,1.4
-abs 2016,poa,abs 2016,sa2,62,8.6
+abs 2016,poa,abs 2016,sa2,62,8.7
 abs 2016,sa1,abs 2016,gcc,100,0.0
 abs 2016,sa1,abs 2016,sa2,100,0.0
 abs 2016,sa1,abs 2016,sa3,100,0.0
@@ -15,7 +15,7 @@ abs 2016,sa2,abs 2016,poa,79,4.3
 abs 2016,sa2,abs 2016,sa3,100,0.0
 abs 2016,sa2,abs 2016,sa4,100,0.0
 abs 2016,sa2,abs 2021,sa2,92,
-abs 2016,sa2,geoscape 202208,postcode,79,
+abs 2016,sa2,geoscape 202211,postcode,79,
 abs 2016,sa3,abs 2016,gcc,100,0.0
 abs 2016,sa3,abs 2016,lga,83,3.0
 abs 2016,sa3,abs 2016,sa4,100,0.0
@@ -35,22 +35,22 @@ abs 2021,sa2,abs 2021,lga,98,
 abs 2021,sa2,abs 2021,poa,84,
 abs 2021,sa2,abs 2021,sa3,100,
 abs 2021,sa2,abs 2021,sa4,100,
-abs 2021,sa2,geoscape 202208,postcode,84,
+abs 2021,sa2,geoscape 202211,postcode,84,
 abs 2021,sa3,abs 2021,gccsa,100,
 abs 2021,sa3,abs 2021,lga,85,
 abs 2021,sa3,abs 2021,sa4,100,
 abs 2021,sa4,abs 2021,gccsa,100,
-geoscape 202208,lga,abs 2016,lga,100,
-geoscape 202208,lga,abs 2021,lga,100,
-geoscape 202208,locality,abs 2016,lga,98,
-geoscape 202208,locality,abs 2016,sa2,94,
-geoscape 202208,locality,abs 2016,sa3,99,
-geoscape 202208,locality,abs 2021,lga,98,
-geoscape 202208,locality,abs 2021,sa2,90,
-geoscape 202208,locality,abs 2021,sa3,99,
-geoscape 202208,locality,geoscape 202208,lga,98,
-geoscape 202208,postcode,abs 2016,lga,93,
-geoscape 202208,postcode,abs 2016,sa3,92,
-geoscape 202208,postcode,abs 2021,lga,94,
-geoscape 202208,postcode,abs 2021,sa3,93,
-geoscape 202208,postcode,geoscape 202208,lga,93,
+geoscape 202211,lga,abs 2016,lga,100,
+geoscape 202211,lga,abs 2021,lga,100,
+geoscape 202211,locality,abs 2016,lga,98,
+geoscape 202211,locality,abs 2016,sa2,94,
+geoscape 202211,locality,abs 2016,sa3,99,
+geoscape 202211,locality,abs 2021,lga,98,
+geoscape 202211,locality,abs 2021,sa2,90,
+geoscape 202211,locality,abs 2021,sa3,99,
+geoscape 202211,locality,geoscape 202211,lga,98,
+geoscape 202211,postcode,abs 2016,lga,93,
+geoscape 202211,postcode,abs 2016,sa3,92,
+geoscape 202211,postcode,abs 2021,lga,94,
+geoscape 202211,postcode,abs 2021,sa3,93,
+geoscape 202211,postcode,geoscape 202211,lga,93,