-
Notifications
You must be signed in to change notification settings - Fork 107
inventory.data.gov
a.k.a Inventory is used by federal agencies to manage metadata for their datasets. Inventory is used to generate the agency's data.json which must be hosted on the agency's website (e.g. agency.gov/data.json). Inventory is a CKAN instance and can be used to host datasets in addition to metadata.
Access to Inventory is historically confusing. There are several mechanisms referring to public/private and each means something different. Since Inventory contains only open data, all data within Inventory can be publicly accessible. However for historical reasons, datasets are only visible to authenticated users while resources are publicly visible.
-
CKAN
private: truea property on the dataset which is seen on the organization datasets listing and affects visibility within CKAN. Only members of an organization can see private datasets within their own organizations. Since Inventory only contains open data and any metadata is published on catalog.data.gov, there is no reason to mark a dataset as private. The CKAN private field is ignored by Inventory's data.json export. -
DCAT-US “public access level” doesn’t mean anything to CKAN and does not affect how the dataset is displayed within Inventory. A dataset with
"accessLevel": "non-public"will still appear in the data.json inventory. - Publishing status (draft or published) is an Inventory concept which affects whether or not the dataset is included in the data.json when exporting from Inventory. It does not affect visibility within CKAN. Any authenticated users will still be able to see draft datasets. Public users be able to see resources of draft datasets. TODO: is this defined in ckanext-datajson?
Resources, the actual data files uploaded to individual datasets, do not have a concept of private and inherit visibility settings from the dataset. Any dataset that includes resource files hosted on Inventory must be marked private: false, otherwise the resource files will not be accessible to anonymous users. This includes some of GSA's hosted datasets that are available by download or the datastore API.
Why are datasets not visible to the public? 🤷 maybe because the confusion around Publishing Status. See GSA/data.gov#2095.
| Instance | Url |
|---|---|
| Production | inventory.data.gov |
| Staging | inventory-stage-datagov.app.cloud.gov |
| sandbox | inventory-dev-datagov.app.cloud.gov |
Sub-components:
- ckan
Services:
- nginx
- PostgreSQL
- redis
- S3
- SOLR
Inventory organizations have a pre-approved list of "publishers" that can be listed in the metadata for a dataset. This list is then uploaded to the inventory organizations, and is used to create a list for the dataset creator. The list is currently here To update the list, you can use the icon in the top right (pencil icon) or clone it and make the necessary changes locally. Once the edits/additions have been made, create a pull request for the changes (similar to https://github.com/GSA/inventory-app/pull/469). Add a team member (or the whole team) as a reviewer. Once the PR is merged, then an automated deployment of this csv occurs (after deployment). You can verify the deployment in GitHub Actions.
ckanpyimport is used in onboarding new agencies to inventory.data.gov. This tool imports datasets from a data.json file.
The import script will happily create duplicates, so if there are any existing datasets in the organization, you probably should delete them all first.
Run this from the jumpbox using nohup or tmux so that disconnecting your session does not interrupt the script. The script can take a while depending on how many packages need to be imported (~2 hours for 1000 datasets). You should also test against staging before running against production.
Once a user has registered via google form and application has been approved by admins, a CKAN user can be added with an Editor role for the agency with the following steps:
- Given user's name and email (lowercase), log into cloud.gov CLI and run cf command:
cf run-task inventory --command "ckan user add firstname-lastname [email protected] password=\'\$(cat /proc/sys/kernel/random/uuid)\'" -k 1500M -m 2G
- Monitor the output and make sure user is successfully created:
# task-name is printed from previous command
cf tail -f -t log inventory| grep some-task-name
- Log into Inventory web UI, go to the Agency's organization page and add user as an Editor.
In case there is an error in step 2 complaining the username is taken, or the email address has been used by another registered user, and you can't find the user on the UI, it means the user is in a deleted state, use the following steps to reactivate it. You need to know the username to do it. If you only know the email address, you will have to run DB queries to find the username.
In case a user exists but in a deleted state, you can use api to reactivate the user, assuming the username is some-user-name.
curl -H "Authorization: <your-token>" -s https://inventory.data.gov/api/action/user_show?id=some-user-name | jq
curl -H "Authorization: <your-token>" -X POST https://inventory.data.gov/api/action/user_patch -d '{"id": "some-user-name", "state": "active"}'
This is configured with cloud.gov, see current configuration as code here.
This should be handled automatically: https://github.com/GSA/inventory-app/blob/main/.profile#L90-L95