Release 3.5.0.-beta: hierarchical import through OAI and CUSTOM interfaces

[aetherfaerber]

We have been experimenting with a custom import interface but more recently switched to an OAI interface. We have been trying to make multi-level imports work with both, but found several obstacles. In case of OAI, these may also be grounded in the standard.

• the import configuration setup page for an OAI interface has neither a field to configure the superordinate ID parameter („ID-Parameter der Überordnung“) nor search fields.
• the import configuration setup page for a CUSTOM interface has a field to configure the superodinate ID parameter but no new search fields may be added, so there is nothing to choose. (Does this imply that the superordinate field should be identical to the subordinate, e.g. “itemID”?)
• We have defined a metadata field as higherlevelIdentifier in the ruleset.xml but choosing import depth 2 during import results in two identical queries for the subordinate ID. What are we missing?
• the parent element type in the OAI interface can only be set to <reference>. What kind of element is this and how can we (and should we?) implement this?

[solth]

* the import configuration setup page for an OAI interface has neither a field to configure the superordinate ID parameter („ID-Parameter der Überordnung“) nor search fields.

This limitation exists because OAI PMH interfaces are not really meant for searching but for harvesting instead and thus do not allow defining custom search fields (including fields querying records by parent ID). To support metadata import from OAI interfaces anyway we make use of the predefined OAI "verb" GetRecord to retrieve individual records, which can not be modified. (see http://www.openarchives.org/OAI/openarchivesprotocol.html#GetRecord for details). If your OAI interface has different search fields defined it may be an indication that it is not a real OAI interface that adheres to the OAI specifications (see comments of #5271, too)

* the import configuration setup page for a CUSTOM interface has a field to configure the superodinate ID parameter but no new search fields may be added, so there is nothing to choose. (Does this imply that the superordinate field should be identical to the subordinate, e.g. “itemID”?)

This is a bug. (and ID parameter and ID parameter for parent record are not supposed to be the same search field.)
The interface type CUSTOM was mainly introduced to enable querying a "fake" OAI interface, see #5257 and the linked PR for details.

Because being able to search for individual records by ID was sufficient for the use case at that time, CUSTOM search interfaces did not allow automatic retrieval of parent records (for which ID parameter for parent record is used) or hit lists (triggered by non-ID search fields). The + button under the search field list which is deactivated after one field has been added, should have a corresponding explanation in it's mouse hover tooltip.

Since enabling these features for CUSTOM type search interfaces would probably require quite a few changes in the code, the best short time solution for this would be to remove the ID parameter for parent record configuration from these search interfaces until more comprehensive changes can be made to add these features, IMHO.

* We have defined a metadata field as higherlevelIdentifier in the ruleset.xml but choosing import depth 2 during import results in two identical queries for the subordinate ID. What are we missing?

This is difficult to say without further investigation. There were several bugs concerning import configurations in release 3.5.0-beta which have since been fixed (see https://github.com/kitodo/kitodo-production/milestone/18 for progress). Perhaps you could try with the most current master branch to see if the problem you described has been resolved as well?

* the parent element type in the OAI interface can only be set to `<reference>`. What kind of element is this and how can we (and should we?) implement this?

This is probably a remnat of earlier version of Kitodo.Production 3 and may not be required anymore. (It should not matter if you select 'reference' here). The corresponding GitHub issue is #5318. I will have to check whether it is still required in some cases when I find the time.

[aetherfaerber]

First of all thank you very very much for pointing me to the relevant issues and the OAI spec. This effort is really appreciated and I will make sure to search and consult the specs next time beforehand.

Alas after going through the spec and the linked issues I don't see my issue resolved but rather manifested in current limitations of the interfaces and hence am back with more questions.

The most important one is ”does this mean hierarchical import is only possible with SRU interfaces but not with OAI or CUSTOM?”

And more explicitely:

This limitation exists because OAI PMH interfaces are not really meant for searching but for harvesting instead and thus do not allow defining custom search fields (including fields querying records by parent ID). To support metadata import from OAI interfaces anyway we make use of the predefined OAI "verb" GetRecord to retrieve individual records, which can not be modified. (see http://www.openarchives.org/OAI/openarchivesprotocol.html#GetRecord for details). If your OAI interface has different search fields defined it may be an indication that it is not a real OAI interface that adheres to the OAI specifications (see comments of #5271, too)

Thank you. After reading the spec I fully agree that OAI interfaces should not have any custom search fields. But we really don't need/want to search (might look like it when used to SRU, though).
Everything we need is described in the spec as “selective harvesting“, namely selective harvesting by set. But this is only available through the ListItems and ListRecords verbs and not the currently implemented GetRecord. Is inclusion of these verbs planned?

Currently, as it seems, the best we could do is to add ListItemsor ListRecords as possible parameter values for parameter key verb as URL parameters for a CUSTOM interface.

But as long as

[…] CUSTOM search interfaces did not allow automatic retrieval of parent records (for which ID parameter for parent record is used) or hit lists (triggered by non-ID search fields).

stays valid I guess… we are out of luck?

Perhaps you could try with the most current master branch to see if the problem you described has been resolved as well?

We switched to the most current master branch but still get the same result for our OAI interface. Although, with the information provided above I am not even sure if this should work at all.

[solth]

Yes, adding support for the OAI verb ListRecords has been on my mind for some time. It could be used to create hit lists similar to those resulting from regular search queries, instead showing all records of the selected OAI set.

This would probably not be a trivial change, though, so implementing this feature will most likely require some time and resources. Perhaps this could be a candidate for the development fund 2023.

Could you provide me with an example URL of the search interface you want to query for an individual record? I could check whether allowing custom search fields for CUSTOM type interfaces is more feasible in the current master or not in order to create the query you need.

[aetherfaerber]

We are going to test the mass import feature but it seems broken (NullPointer Exception) in the beta release. So we will have to make sure to switch to the branch you mentioned above first.

Can you elaborate on that? What have you tried and where exactly does the NPE happen?

First of all sorry the very belated reply. I was mostly unable to work in February.

Regarding the question about NPEs sadly I cannot reproduce this behaviour now, although we are quite sure that nothing has changed in our setup since I reported it.
I tried really hard to get a NPE yesterday and succeeded once after going directly to massimport.jsf and starting imports without a production template but I cannot reproduce this now either and am quite sure this is not what I did a month ago.

[aetherfaerber]

I have marked this one as answer since it contains the pointer towards the mass-import feature, which we finally managed to use for hierarchical imports.

Also, i will make sure to submit adding support for ListRecords as a candidate for the development fund.

[solth]

We are going to test the mass import feature but it seems broken (NullPointer Exception) in the beta release. So we will have to make sure to switch to the branch you mentioned above first.

I was unable to reproduce a NullPointerException on the mass import page with the current state of the code, but there does seem to be one instance of unhandled exception in form of a ConfigException (documented in #5584). Could this maybe the same error you encountered?

[aetherfaerber]

Also, i will make sure to submit adding support for ListRecords as a candidate for the development fund.

I have opened an issue: #5587
@solth could you please add the Development fund label to the issue?

[solth]

@solth could you please add the Development fund label to the issue?

done!

[BartChris]

[…] CUSTOM search interfaces did not allow automatic retrieval of parent records (for which ID parameter for parent record is used) >> or > hit lists (triggered by non-ID search fields).

stays valid I guess… we are out of luck?

In addition to everything said above: It appears to me that in your case the same field ("identifier") is used for retrieving children and parent records. So the automatic retrieval of a parent record should also work (at least for the normal process creation). I am also using a custom interface and when i import a child record the parent record is also imported. So this works for me.

Edit: But i am also not sure WHY this works - i am still not fully sure about the exact purpose of the functional keys higherlevelidentifier, recordIdentifer and the ID parameter for parent record in the context discuessed here. For retrieving a parent record which is already in the database higherlevelidentifier and recordIdentifer are used. But i am not sure right what exactly is used for creating a new parent record from an imported child. @solth knows best about this, so he can probably answer that.

[aetherfaerber]

Thank you very much for the additional information @BartChris .

Regarding a clarification of the functional keys, the only documentation I know of is a comment in the ruleset file. It at least marks the purpose of higherlevelIdentifier:

Description of the applied attributes:
1. processTitle="TSL_ATS+'_'+CatalogIDDigital" -> Definition of the process title.
2. use="createChildrenFromParent" -> For document types like periodical, when children proce    sses should be created by the parent process in the process list.
3. use="createChildrenWithCalendar" -> For document types like newspaper, when the calendar     should be available for parent processes in the process list to create processes for issues.
4. use="higherlevelIdentifier" -> For metadata which contains the identifier of the superordinate process.
5. use="processTitle" -> For document types which are created with the calender without Cata    logIDDigital, like newspaper issues. The process title can be transformed in the export XSL.
6. withWorkflow="false" -> For superordinate document type like periodical, newspaper, multivolume works, whose processes do not need a workflow.

This does help with assigning the keys but doesn't explain anything about the inner workings and interrelations. As you said, a further explanation would be appreciated.

The automatic import of a parent record (which isn't our use case but could be if we can mass-import all children and they get automatically assigned to only one singe parent process) didn't work for us last time i checked, but I will try this again in order to report back.

[BartChris]

My understanding of those keys is that higherlevelidentifier and recordIdentifer are attributes which you assign to keys defined in your ruleset. The higherlevelidentifier marks the field which contains the key of the parent record of the current record. The recordIdentifier marks the key which contains the identifier of the digital object in the source metadata system.

The ID parameter for parent record is not about keys in your ruleset but about search fields in an external search interface. You basically define the key which is used to search in all records of an external catalogue for those titles which have a link to a specific parent record. You need that key if you want to import all child records for a specific parent record from a catalog.

So when the system is searching if there is already an parent record in the database for the newly created record the following is happing:

1. the ID inside the field marked as higherlevelidentifier is extracted; this is the identifier of the parent object we want to search in our Kitodo database/Kitodo search index
2. This ID is then searched for in all processes and there specifically in all fields which are marked as recordIdentifier fields (which could be different fields for different document types):

https://github.com/effective-webwork/kitodo-production/blob/6c8240351c764e5838a7bc66afa58d0399615b7f/Kitodo/src/main/java/org/kitodo/production/services/data/ImportService.java#L934-L960

When the parent's record identifier is not already contained in the database, Kitodo is trying to import the parent from an external catalog. The number of times this import logic is called depends on the number of import levels defined during import. This is done using repeated/recursive calls of the method importProcessAndReturnParentID with supplying the current parentid as recordid which now should be imported,

The logic works in a way that after the import of a record, the higherlevelidentifier of this record is extracted using the defined value for higherlevelidentifier and then saved as local variable parentid

https://github.com/effective-webwork/kitodo-production/blob/6c8240351c764e5838a7bc66afa58d0399615b7f/Kitodo/src/main/java/org/kitodo/production/services/data/ImportService.java#L376-L392

This parentid can then be used to query the search interface for a parent record with this ID reusing the normal import logic. This implies in my understanding that the parent record and the child record need to use the same field in the metadata source to store their respective identifier, which is then queried.

https://github.com/effective-webwork/kitodo-production/blob/6c8240351c764e5838a7bc66afa58d0399615b7f/Kitodo/src/main/java/org/kitodo/production/services/data/ImportService.java#L544

[aetherfaerber]

Thank you for these explanations. They really gave us a better understanding of what Kitodo is doing here. Also your explanation of recordIdentifier was what we needed to finally get mass imports working.