Request for feedback: Proposed monthly data file snapshot service

[codycooperross]

A proposed service to enhance metadata harvesting

Based on input from DataCite community members, we are beginning to outline an initial feature set for a monthly data file snapshot service that aims to provide easier and more efficient mechanisms for metadata harvesting. Review the proposed initial feature set below and let us know what you think!

Proposed initial feature set

1. A complete DataCite data file snapshot, updated monthly with metadata records separated by month updated.
   1. The snapshot would contain a record of every Findable state DOI, each represented once in a folder corresponding to the month the DOI was last updated. The entire snapshot would be regenerated monthly.
   2. A series of compressed files in each folder would contain records updated during the month. The format of individual records would reflect a DOI singleton response from the DataCite REST API minus the xml value, and each compressed file would contain up to 10,000 of these records in JSON Lines format.
   3. Users could download the entire data file to bootstrap their metadata harvesting or only retrieve newly updated records from the most recent month(s).
2. Tabular reports (CSV) that list all DataCite DOIs, the date they were updated, their current state (Findable or Registered), and their current Repository ID (client-id).
   1. Each CSV file would list all Findable and Registered state DOIs updated during the month indicated in the file and folder names. Registered state DOIs would be included to identify DOIs that have been removed. Repository IDs would aid in identifying DOIs that have moved between repositories.

The directory structure would look something like this:

dois
├── updated_2024-07
│   ├── 2024-07.csv
│   ├── part_0000.jsonl.gz
│   ├── part_0001.jsonl.gz
│   ├── part_0002.jsonl.gz
├── updated_2024-08
... ├── 2024-08.csv
    ├── part_0000.jsonl.gz
    ├── part_0001.jsonl.gz
    ├── part_0002.jsonl.gz
    ├── part_0003.jsonl.gz
    ...

Rows in the CSV file would be structured like the following:

doi	state	client_id	updated
10.0166/fk2.stagefigshare.9753211	findable	figshare.dud	2024-08-01T00:20:45Z
10.15493/dea.mims.20210317	registered	nrf.saeon	2024-08-01T00:32:57Z

Providing feedback

We are interested in your thoughts on these initial features and possibilities for improvement. Contribute your feedback by leaving a comment on this post. You are welcome to reply by answering any or all of the questions below:

• Would you use the features described above?
• Are there any features that you would change or that may be difficult to adopt?
• What would be the best way for you to access and download the data file?

Note

We are also seeking feedback on the annual DataCite Public Data File. You can share feedback here: #133.

This discussion will close on 13 December 2024, but you can still submit feedback on DataCite Suggestions.