Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test and tweak strapi import for different types (RPB-58) #37

Merged
merged 35 commits into from
Sep 21, 2023
Merged
Changes from 1 commit
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
8111944
Test and tweak strapi import for `articles` (RPB-58)
fsteeg Jun 26, 2023
e7d4666
Merge remote-tracking branch 'origin/rpb-28-hbzIds' into rpb-58-types
fsteeg Jun 27, 2023
835cc15
Update README for new unified content type (RPB-58)
fsteeg Aug 8, 2023
70902b6
Merge remote-tracking branch 'origin/main' into rpb-58-types
fsteeg Aug 9, 2023
f9ee6f6
Create title (f20_) and dates (f76a, f76b) for volumes (RPB-58)
fsteeg Aug 9, 2023
f84d5ff
Add test data for Strapi import of monographs (RPB-58)
fsteeg Aug 10, 2023
d6b8728
Tweak rpb-titel-to-strapi.fix for URL and Signatur (RPB-58)
fsteeg Aug 10, 2023
0c8857d
Tweak transformations for full data processing (RPB-58)
fsteeg Aug 10, 2023
5c735bb
Set up workflow for full Strapi import (RPB-58)
fsteeg Aug 10, 2023
0addbaa
Transform `f18_=x` to boolean for Strapi (RPB-58)
fsteeg Aug 11, 2023
a1fa272
Validate URLs, extract macro, add list with invalid URLs (RPB-58)
fsteeg Aug 11, 2023
6173d3f
Add list of monographs (f36_=s) with missing 76a / 76b (RPB-58)
fsteeg Aug 14, 2023
564140d
Tweak handling of required `f70s` pages field (RPB-58)
fsteeg Aug 15, 2023
a4010ff
Handle both upper and lower case 'x' for `f18_` (RPB-58)
fsteeg Aug 15, 2023
521c02f
Use `log-object` in import workflows for improved logging (RPB-58)
fsteeg Aug 16, 2023
2489480
Remove custom `wait` command (RPB-58)
fsteeg Aug 16, 2023
69849e0
Restrict 70 transformation to 701 etc., not 70b etc. (RPB-58)
fsteeg Aug 17, 2023
326b752
Remove printing of records with missing page numbers (RPB-58)
fsteeg Aug 17, 2023
94f494c
Set required `f70` if missing, point to specific fields (RPB-58)
fsteeg Aug 17, 2023
447bc81
Always set `f18_` to true if the field exists (RPB-58)
fsteeg Aug 18, 2023
2b16d02
Set required f70 field only for articles (RPB-58)
fsteeg Aug 18, 2023
92e1952
Don't try to fill f76[ab] from f01_, point to it instead (RPB-58)
fsteeg Aug 22, 2023
a17c861
Add `sm` / `sbd` examples, update README on Strapi import (RPB-58)
fsteeg Aug 23, 2023
45a8386
Update SW lookup table for current data
fsteeg Aug 30, 2023
78033ab
Create URIs for Sach- (30) and Ortsnotation (31) in Strapi (RPB-58)
fsteeg Aug 30, 2023
bd0da0a
Move complex subject splitting and URI creation to Strapi (RPB-58)
fsteeg Aug 30, 2023
bbefc0b
Move URI creation for contrib. agents and roles to Strapi (RPB-58)
fsteeg Aug 31, 2023
9c968b3
Group numbering with superordinate link in `f01` object (RPB-58)
fsteeg Sep 1, 2023
bfe3eb2
Rename `ü` subfields to `u` for Strapi compatibility (RPB-58)
fsteeg Sep 1, 2023
65c4e08
Remove handling of `f36t` from to-lobid transformation (RPB-58)
fsteeg Sep 4, 2023
27abd36
Handle unsupported repetition of `f01_` by picking first (RPB-58)
fsteeg Sep 4, 2023
b1934aa
Rename `id` subfield for Strapi compatibility (RPB-58)
fsteeg Sep 4, 2023
8250a0d
Remove `f01.f01s` if it's not a year, i.e. 4 digits (RPB-58)
fsteeg Sep 5, 2023
d948077
Add monograph without publication dates to sample data (RPB-58)
fsteeg Sep 5, 2023
e71b261
Add manually corrected 036t0115179 to test data (RPB-58, RPB-69)
fsteeg Sep 6, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove handling of f36t from to-lobid transformation (RPB-58)
And from content type, since we won't catalog periodicals in Strapi
fsteeg committed Sep 4, 2023
commit 65c4e08aca4d82b220068f13dcae7591eabe0477
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -35,7 +35,7 @@ This writes a `.tsv` file to `output/`, to be used for lookups in the transforma
sbt "runMain rpb.ETL conf/rpb-test-titel-to-strapi.flux"
```

This writes a single `.json` files to `output/` (it's actually JSON lines, but the suffix makes it work with JSON tools, e.g. for syntax coloring and formatting).
This writes a single `.json` file to `output/` (it's actually JSON lines, but the suffix makes it work with JSON tools, e.g. for syntax coloring and formatting).

### Import strapi data

2 changes: 1 addition & 1 deletion conf/output/test-output-56.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"@context" : "http://lobid.org/resources/context.jsonld",
"type" : [ "BibliographicResource", "Periodical" ],
"type" : [ "BibliographicResource", "MultiVolumeBook" ],
"rpbId" : "RPB121t0144215",
"id" : "https://lobid.org/resources/RPB121t0144215",
"inCollection" : [ {
6 changes: 1 addition & 5 deletions conf/rpb-titel-to-lobid.fix
Original file line number Diff line number Diff line change
@@ -20,11 +20,7 @@ if all_equal("f36_", "s")
set_field("type[].$append", "Book")
end
if any_equal("f36_", "sm")
if exists("f36t")
copy_field("f36t", "type[].$append")
else
set_field("type[].$append", "Periodical")
end
set_field("type[].$append", "MultiVolumeBook")
end
if any_equal("f36_", "sbd")
set_field("type[].$append", "Book")