Skip to content

Latest commit

 

History

History
281 lines (168 loc) · 12 KB

Detailed_Documentation.md

File metadata and controls

281 lines (168 loc) · 12 KB
Raw

NOTICE: It's better to navigate the folders with ArcGIS. Because some data such as shapefile contains .cpg, .dbf, .shp, .xml, .prj and so on but they should be considered as a single dataset.

Directory: Z:\Research\dfuller\Walkabilly\exposure\transit

CT and DA folder contains census tract and dissemination area data of Canada. The topologyCTDA.gdb and TopologyCTDA.mxd are used to check the topology correctness of CT and DA data. There is no topology error in the dataset.

PRUID Province or territory name Abbreviation
10 Newfoundland and Labrador/Terre-Neuve-et-Labrador NL
11 Prince Edward Island/Île-du-Prince-Édouard PE
12 Nova Scotia/Nouvelle-Écosse NS
13 New Brunswick/Nouveau-Brunswick NB
24 Quebec/Québec QC
35 Ontario ON
46 Manitoba MB
47 Saskatchewan SK
48 Alberta AB
59 British Columbia/Colombie-Britannique BC
60 Yukon YK
61 Northwest Territories/Territoires du Nord-Ouest NT
62 Nunavut NU

The following provinces or territories do not have Census Tract data: PE, YK, NT, NU

Each province/territory has a folder named as "PTAI_[Abbreviation]", for example, for Alberta, the folder is "PTAI_AB". Each PTAI folder has the same structure. Inside the PTAI folder, there is a folder called "input", it contains all the input data for the python script. These input data are intermediate data generated from "CT and DA" data and the census data downloaded from Statistics Canada.

Inside the "input" folder, there is a file geodatabase named with the abbreviation of the province/territory. For example, the geodatabase for Alberta is named "AB.gdb". The geodatabase contains both CT and DA data with NAD 83 North_America_Equidistant_Conic projected coordinate system and NAD 83 North_America_Albers_Equal_Area_Conic projected coordinate system. The data with equal area projection is used to calculate the area of CT and DA.

Data preprocessing steps:

  1. Change the projection of CT and DA data from Lambert_Conformal_Conic to North_America_Albers_Equal_Area_Conic

Input:

lct_000a16a_e.shp

Output (feature classes in geodatabase):

CT_CA_EqA (North_America_Albers_Equal_Area_Conic)

Method:

Project (Data Management)

  1. Create CT data for each province/territory

Input:

CT_CA_EqA

Output (feature classes in geodatabase):

CT_AB_EqA, CT_BC_EqA, CT_MB_EqA…and so on

Method:

  1. Open attribute table, select by attributes

  2. Click "PRUID", click "Get Unique Values"

  3. Type an SQL query "PRUID = 48" (here 48 represents AB, the PRUID corresponding province can be found above), click apply

  4. Right click the layer and click "Data -> Export Data", then save the selected data as a new future class in the geodatabase

  5. Repeat the steps for other provinces/territories

  6. Create DA data for each province/territory

Similar to CT

Input:

DA_CA_EqA

Output (feature classes in geodatabase):

DA_AB_EqA, DA_BC_EqA, DA_MB_EqA…and so on

Method:

  1. Open attribute table, select by attributes

  2. Click "PRUID", click "Get Unique Values"

  3. Type an SQL query "PRUID = 48" (here 48 represents AB, the PRUID corresponding province can be found above), click apply

  4. Right click the layer and click "Data -> Export Data", then save the selected data as a new future class in the geodatabase

  5. Repeat the steps for other provinces/territories

  6. Create population data table for each province/territory

Input:

PopulationCanada.csv

Output:

Population_AB.xlsx

Population_AB (standalone table in geodatabase)

Method:

  1. Select all the records and click "Format As Table"

  2. In "PRUID" column, select "PRUID = 48" (here 48 represents AB province, the PRUID corresponding province can be found above)

  3. Copy the data and paste it to Population_AB.xlsx file

  4. In ArcMap, use "Excel to Table" to create a standalone table "Population_AB" in the geodatabase

  5. Repeat the steps for other provinces/territories

  6. Table join between DA data and population data

Input:

DA_AB_EqA

Population_AB

Output:

DA_AB_EqA (with population information in the attribute table)

Method:

  1. Notice that the "geographic code" in the table is "DAUID" in the DA layer, but their data types are different, so add a new field "DAUID" in the population data table

  2. Calculate the value of "DAUID" using "Field Calculator", type "DAUID = Geographic_code"

  3. Right click the layer select "Joins and Relates-> Join"

  4. Select "Join attributes from a table", then choose the field in this layer that the join will based on "DAUID", choose the "DAUID" field in "Population_AB" table.

  5. Repeat the steps for other provinces/territories

  6. Create a summary table to calculate the population in each CT

Input:

DA_AB_EqA (with population information in the attribute table)

Output:

Population_AB_CT (Standalone table in geodatabase)

Method:

  1. Open the attribute table of DA_AB_EqA

  2. Highlight "CTUID" field, click "Summarize…" (NOTICE, this step cannot use "CTNAME"!!!)

  3. Choose "Population_NL.Population_2016" -> "Sum" to be included in the output table

  4. The output table contains the population information for each CT

  5. Repeat the steps for other provinces/territories

  6. Table join between CT_AB_EqA and Population_AB_CT

Input:

CT_AB_EqA

Population_AB_CT

Output:

CT_AB_EqA (with population information in the attribute table)

Method:

  1. Right click the layer select "Joins and Relates-> Join"

  2. Select "Join attributes from a table", then choose the field in this layer that the join will based on "CTNAME", choose the "CTNAME" field in "Population_AB_CT" table.

  3. Repeat the steps for other provinces/territories

  4. Calculate the population density for CT

Input:

CT_AB_EqA (with total population for each CT)

Output:

CT_AB_EqA (with population density for each CT)

Method:

  1. Open attribute table of CT_AB_EqA, in "Table Options", select "Add Field…"

  2. Add two fields, one of them is "Area_in_km2", another is "PopD_CT", both of their types are "double"

  3. For the "Area_in_km2", use "Calculate Geometry…" to calculate the area, units in square kilometers.

  4. For PopD_CT, use "Field Caculator…". The formula is "PopD_CT = Sum_population_2016/Area_in_km2"

  5. Repeat the steps for other provinces/territories

  6. Change the projection of CT and DA data from North_America_Albers_Equal_Area_Conic to North_America_Equidistant_Conic

Input:

CT_CA_EqA (North_America_Albers_Equal_Area_Conic)

Output (feature classes in geodatabase):

CT_CA_EqD (North_America_Equidistant_Conic)

Method:

Project (Data Management)

  1. Create PCCF point feature

Input:

pccfNat_fccpNat_062017.txt

Output:

PCCF point feature for each province

Method:

  1. Since the .txt file has too many records to convert to .csv file directly, an R script is used to divide the .txt file based on the first letter of the post code. The outputs are PCCF_AB.txt, PCCF_BC.txt and so on.

  2. Open Excel, export the PCCF_AB.txt to .csv file. "Data" -> "Get Data From Text", create break lines to separate each field. Save file as PCCF_AB.csv

  3. In ArcMap, "File" -> "Add Data" -> "Add XY Data". Choose PCCF_AB.csv, set longitude value as X field, latitude value as Y field, and coordinate system as "WGS 84".

  4. Use "Project" management tool to project the point feature to North_America_Equidistant_Conic coordinate system. The outputs for this step are PCCF_AB_EqD and so on

  5. Create CT and DA data for cities

Input:

CT and DA data for provinces (for example: CT_AB_EqD)

Output:

CT and DA data for provinces (for example: CT_Calgary_EqD.shp)

Method:

Select by attribute (CMANAME = "Calgary"), then export the selected data

  1. Clip PCCF data to city boundaries

Input:

PCCF data for provinces

Output:

PCCF data for cities

Method:

Use "Clip" tool, choose PCCF_AB_EqD as Input Features, CT_Calgary_EqD as Clip Features. After creating the PCCF data for cities, one more step is to create a "pccfID" field for it. In the attribute table, "Add Field", then "Field Calculator…", "pccfID = FID"

  1. Create road network dataset for cities

Input:

CanVec Data of Canada (for example: canvec_50K_AB_Transport_shp/canvec_50K_AB_Transport/road_segment_1_1.shp)

Output:

Roads network dataset for cities

Method:

  1. Project the input data to North_America_Equidistant_Conic coordinate system

  2. Clip the roads segment data using CT for cities boundaries, get roads line feature for cities

  3. In Catalog, right click the roads line feature, click "New Network Dataset", and notice that the network dataset can be only created based on shapefile.

  4. Now the input data for the script are ready and it is inside the "input" folder of each city folder. The input data is: CT, DA, PCCF and Roads network. All of them have the same coordinate system -- North_America_Equidistant_Conic.

An example is:

CT_Calgary_EqD.shp

DA_Calgary_EqD.shp

PCCF_Calgary_EqD.shp

Roads_Calgary_EqD.shp

Roads_Calgary_EqD_ND.nd

Roads_Calgary_EqD_ND_Junctions.shp

  1. Download the transit feed (GTFS) data for each city
  2. Run the script. After running the script, an intermediate folder is created, and all the intermediate data is inside the folder. The PTAI for CT and DA are created
  3. Copy the PTAI for CT and DA from the geodatabase to shapefiles in a new "output" folder.
  4. Create map for each city with CT and DA level.

Trouble Shooting:

  1. The script for each city are slightly different in terms of the input name. Before running the script, make sure replace the old city name with the new city name. There are 4 file names need to be replaced.
  2. In ArcGIS, when a process accesses a dataset, it locks the dataset, preventing other processes from changing it. Therefore, when run the script, it is important to close all the related dataset in ArcGIS, or even close ArcGIS itself. Due to the same reason, sometimes a folder/gdb/file cannot be deleted. In case of this, ArcGIS needs to be closed.
  3. If we add a field to a table, but the field already exists, there will be error. Thus if we run the script more than one time, it is important to make sure the original input files are not changed. Specifically, check whether the "EF" filed has been deleted from PCCF data table.
  4. Gtfs data needs to be checked, for example the format of gtfs data of red deer is not suitable for the script, so I downloaded another dataset.
  5. Previously, the output of Regina area has too many 0s. That's because in the script, several fields have been deleted thus when run the join management, the output of point feature only contains 1 point
  6. When run the OD Matrix Analysis, if the mode is empty, there will be an error. For example, Calgary, Edmonton and Waterloo routes only have mode 0 (Tram, Streetcar, Light rail) and mode 3 (bus). So we need to skip the mode 1 and mode 2 when running the OD Matrix Analysis.
  7. For the map document files (.mxd), it's better to store the relative pathnames. In ArcGIS, "File" -> "Map Document Properties" -> "Pathnames", check "Store relative pathnames to data sources". Thus when the map document file moves together with the source files (PTAI for DA and CT), the link won't be broken. In case the link is broken, that means the document cannot find the source file, then we have to click the layer properties, find "source" tab and set data source.