Adding export.socrata function #187
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Save data.json to file system ------------------------------ A copy of the data.json file at the beginning of the download process is saved alongside the actual downloaded data. Since `export.socrata()` uses data.json as the index to download data, this will allow users to cross-reference the downloaded data with other metadata associated with it available through [Project Open Data](https://project-open-data.cio.gov). Handle non-data file --------------------- Socrata lists non-data files, such as Socrata Stories--HTML websites that contain text but no machine-readable data--in the data.json file. This causes errors when trying to download those sites because they do not have a "distribution URL". While it's arguable that these "sites" should not be included in the first place, the script now simply skips those files. Since a copy of the data.json file is downloaded (see above), users will have transparency into which URLs were not downloaded.
Socrata supports external links which direct to web pages (e.g., HTML). These would cause an error when `export.socrata()` attempted to download them. This fix will simply skip those files and proceed to the next file.
* Ignores HTML files (e.g., Socrata Pages) * Ignores on occassions there isn't any data * Will download (uncompressed) PDFs, Word, Excel, PowerPoint, plain text attachments.
Rebased branch with most recent `dev` branch and generated documentation. Merge branch 'dev' into issue126 # Conflicts: # DESCRIPTION # R/RSocrata.R
* Removed user-defined option for file output (not available yet) * Clarified documentation where `export.socrata()` files will be located. * Fixed incorrect date in `DESCRIPTION` file. * Iterating build number.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces a stable version of an
export.socrata()function as outlined in #126. This allows users to download the contents of a data portal to a local directory. This function will download CSVs (compressed), PDFs, Word, Excel, PowerPoints, GeoJSON, Shapefiles, plain text documents (uncompressed), etc. It will not download HTML pages. As part of the process, the function also copies thedata.jsonfile to act as an index for other downloaded files.I've proposed the version as 1.8.0.
Testing portal export
To test this function, I used the City of Norfolk, VA to export all of the data sets. Looking at their data.json file, I counted 32 data sets that were not HTML pages or did not have a downloadable file. Executing
export.socrata("https://data.norfolk.gov)resulted in 32 downloaded files plus the copy of thedata.jsonfile. Thus, the expected number of files match the actual number of downloaded files.Testing non-CSV documents
All of the testing for Norfolk resulted in compressed CSV files, however, also needed to test the ability to download non-CSV files. Kansas City, Missouri's data portal has an unusually large number of non-CSV data sets on their portal, such as PDFs, word documents, Excel documents, etc.
I tested the function on downloading files from their data portal. The function downloaded PDFs, Words, Excel, and other non-CSV files along with CSV files.
However, I did encounter frequent network timeouts after approximately 80 items were downloaded. I believe this is limited to the network and not an issue with the function itself. While this may not be a bug, it may be a limitation on the ability to export files from Socrata.
Unit Testing
I have not written a unit test. I think any unit test will take too much time and space for typical unit testing. The smallest portal download, Norfolk, elapsed over 30 minutes to complete all downloads.
In general, a recommended method for testing is to choose a reasonably small portal and do the following:
*
distribution/mediaTypeis blank*
distribution/mediaTypeistext/html*
distribution/downloadURLis blankIdeally, the portal being used to test contains CSV files as well as non-CSV files.