Data source problem: the new dataset has duplications and misses some documents

[andrew2net]

The new dataset has 417 duplications. In each duplication case, the URLs of duplicated documents are identical.
Here are the URLs of duplicated docs:
log.txt

Originally posted by @andrew2net in #112 (comment)

[andrew2net]

Also, the MODS dataset has 193 docs less than allrecords.xml. Also, some docs exist only in the MODS dataset, so more than 193 are missed. Here is the list of the differences diff.txt

• Lines that are only in the old dataset are prefixed with <.
• Lines that are only in the new dataset are prefixed with >.
• If a file name has changed, it will be shown as the file from the old dataset and then the file from the new dataset.
• Each change is preceded by the line numbers that apply to that change. The format is start,end for ranges of lines, or just a single number for single lines. The line numbers for the first file and the second file are separated by c (for changed), a (for added), or d (for deleted).

[ronaldtse]

@andrew2net in the diff file I see a number of "IPD" entries.

I believe only the NIST CSRC source provides IPDs and other draft entries and the CSWPs. These were never given by the NIST-Tech-Pubs repository. Can you update the diff file? Thanks.

[andrew2net]

@ronaldtse do you mean NIST IR 8320C ipd? There were such documents in allrecords.xml

   ...
   <query key="IR">
      <doi type="report-paper_title">10.6028/NIST.IR.8320C.ipd</doi>
      ...
   <query key="IR">
      <doi type="report-paper_title">10.6028/NIST.IR.8286D.ipd</doi>
      ...

[ronaldtse]

Very interesting. I will report this to NIST.

[ronaldtse]

The new dataset has 417 duplications. In each duplication case, the URLs of duplicated documents are identical. Here are the URLs of duplicated docs: log.txt

We now have a command that diffs the duplicated docs to see what happened:

• Add detect-duplicates command loc_mods#9

I have posted the detailed diff to NIST here:

• MARC/MODS data source: Duplicated records in MODS database but with differential data usnistgov/NIST-Tech-Pubs#44

[ronaldtse]

@andrew2net the "missing documents list" is inaccurate.

< NBS_FIPS_11-1-SEP30_1977.yaml
---
> NBS_FIPS_11-1-SEP30.yaml
< NBS_FIPS_89-SEP1.yaml
---
> NBS_FIPS_89-SEP1981.yaml

These IDs are wrong. I have reported them:

• MARC/MODS data source: NBS.FIPS.89-May1981 and NBS.FIPS.89-Sep1981 PubID incorrect usnistgov/NIST-Tech-Pubs#51
• MARC/MODS data source: PubIDs of NBS.FIPS.11-1-SEP30 and NBS.FIPS.11-1-SEP1977 incorrect usnistgov/NIST-Tech-Pubs#50

I have investigated these IDs and reported many issues to NIST:

• MARC/MODS data source: Duplicated records in MODS database but with differential data usnistgov/NIST-Tech-Pubs#44
• MARC/MODS data source: New LCIRC 1040 DOI unreachable usnistgov/NIST-Tech-Pubs#45
• MARC/MODS data source: New NIST.IR.85-3273-14r DOI unreachable (correct ID, but DOI not working) usnistgov/NIST-Tech-Pubs#46
• MARC/MODS data source: NIST GCR 80-287 DOI (https://doi.org/10.6028/NIST.GCR.80-287) unreachable usnistgov/NIST-Tech-Pubs#47
• MARC/MODS data source: Metadata and DOI for "NIST HB 105-3-1997" incorrect, should be "NIST HB 105-3r1997" usnistgov/NIST-Tech-Pubs#48
• MARC/MODS data source: "NIST SP 2200-05" listed as "NIST SP 2100-####" usnistgov/NIST-Tech-Pubs#49
• MARC/MODS data source: NCSTAR documents now all mislabeled as NIST.MONO in DOI? usnistgov/NIST-Tech-Pubs#52
• MARC/MODS data source: 269 missing records compared to previous allrecords.xml from CrossRef metadata usnistgov/NIST-Tech-Pubs#53