Export Import Data

Copy data from one instance to another through raw data export / import

To copy data for a specific survey from one instance to another, these steps can be followed:

• copy survey
• export a raw data report for the specific survey in the source instance
• export an (empty) raw data report for the copied survey in the destination instance
• manually copy the data from the source report to the destination report, taking care that the data goes under the same column headings.
• DO NOT copy the first and last column, which contains the surveyInstanceID and the MD5 hash of the data, respectively.
• reimport the report in the destination instance.

The original raw data report cannot be used because the Ids of the questions can be different in the copied survey.

Copy bulk data using appcfg.py

System Requirements

• Python >= 2.5
• Python GAE SDK
• A working FLOW instance your local machine
• A clone of akvo-flow-data-config repo

Instructions

Linux

• Download and extract the Python GAE SDK
• Copy the csv_fix.py to the main folder of SDK
• Use the appcfg.py script to perform actions

Mac OS X

• Download and extract the Python GAE SDK
• This will put a GoogleAppEngineLauncher.app in your Applications folder
• Copy the csv_fix.py to the main folder of the SDK To go to this folder, you will need to right-click on the GoogleAppEngineLauncher.app and select "Show Package Contents":
• Set up a proxy by putting the following in my httpd-vhost.conf file (in etc/apache2/extra/):

    <Location /remote_api>
      ProxyPass http://localhost:8888/remote_api
    </Location>

• Note: 8888 is the port where your FLOW instance is running
• Then run: sudo apachectl restart to restart the apache server.

Examples

Uploading data to a local dev machine

$ ../google_appengine/appcfg.py upload_data \
  --kind=Geometry --file=Geometry.csv \
  --url=http://localhost:8888/remote_api \
  [email protected] --config_file=config.yml 

Where:

• ../google_appegine is the path where the SDK is located
• --url=http://localhost:8888 is the URL where the FLOW instance is running
• upload_data / download_data is the action you want to perform

Downloading data from a remote GAE instance

$ for i in $(awk '/^- kind/ {print $3}' config.yml); do ../google_appengine/appcfg.py download_data \
  --config_file=config.yml \
  --file=$i.csv --kind=$i \
  --url=http://flowdemoenvironment.appspot.com/remote_api \
  --application=s~flowdemoenvironment \
  [email protected]; done;

Where:

• application is the id of the GAE app, usually the same of the url
• email is a valid email with admin rights to that instance

Upload of all entities to a local dev machine

WARNING: This could take several minutes/hours depending on your dev machine (RAM/CPU)

$ for i in $(awk '/^- kind/ {print $3}' config.yml); do ../google_appengine/appcfg.py upload_data \
  --config_file=config.yml \
  --file=$i.csv --kind=$i \
  --url=http://localhost:8888/remote_api \
  [email protected] --passin < /path/to/fake/password.txt; done;

• password.txt contains a string with any password

More info

• http://bulkloadersample.appspot.com/