inventory.data.gov

a.k.a Inventory is used by federal agencies to manage metadata for their datasets. Inventory is used to generate the agency's data.json which must be hosted on the agency's website (e.g. agency.gov/data.json). Inventory is a CKAN instance and can be used to host datasets in addition to metadata.

Access

Access to Inventory is historically confusing. There are several mechanisms referring to public/private and each means something different. Since Inventory houses open data, all data within Inventory should be publicly accessible. However for historical reasons, datasets are only visible to authenticated users while resources are publicly visible.

• CKAN private: true a property on the dataset which is seen on the organization datasets listing and affects visibility within CKAN. Only members of an organization can see private datasets within their own organizations. Since Inventory only houses open data, there is never a reason to mark a dataset as private.
• DCAT-US “public access level” doesn’t mean anything to CKAN and does not affect how the dataset is displayed within Inventory. A dataset with "accessLevel": "non-public" will still appear in the data.json inventory.
• Publishing status (draft or published) is an Inventory concept which affects whether or not the dataset is included in the data.json when exporting from Inventory. It does not affect visibility within CKAN. Any authenticated users will still be able to see draft datasets. Public users be able to see resources of draft datasets. TODO: is this defined in ckanext-datajson?

Resources, the actual data files uploaded to individual datasets, do not have a concept of private and inherit visibility settings from the dataset. Since almost all datasets are open data and marked private: false, nearly all resources in Inventory are visible to the public.

Why are datasets not visible to the public? 🤷 maybe because the confusion around Publishing Status.

Environments

Instance	Url
Production	inventory.data.gov
Staging	inventory-datagov.dev-ocsit.bsp.gsa.gov
sandbox	inventory.sandbox.datagov.us

Dependencies

Sub-components:

• ckan
• datapusher

Services:

• apache2
• rds
• redis
• s3
• solr

Logs

• /var/log/inventory/ckan.custom.log
• /var/log/inventory/ckan.error.log
• /var/log/inventory/datapusher.custom.log
• /var/log/inventory/datapusher.error.log

Common tasks

Importing from data.json

ckanpyimport is used in onboarding new agencies to inventory.data.gov. This tool imports datasets from a data.json file.

The import script will happily create duplicates, so if there are any existing datasets in the organization, you probably should delete them all first.

Run this from the jumpbox using nohup or tmux so that disconnecting your session does not interrupt the script. The script can take a while depending on how many packages need to be imported (~2 hours for 1000 datasets). You should also test against staging before running against production.

Alert conditions