Pagingation in IDS

[kragall]

When Connectors query data from the Clearing House, they can either query for a single stored document or for all documents that are stored under a process_id. In the latter case this can result result in problems with too large responses, e.g. long response times or very large response messages.

In order to avoid these issues the Clearing House offers the Connectors the possibility to add query parameters to the Query-Endpoint, e.g. "?page=1&size=100", which will result in the Clearing House sending responses with at most 100 stored documents. With this solution arises the issue that the Connector needs to know, either what page is the last page that exists, or how many documents are stored in total.

A possible solution of this issue would be the introduction of a PagingResultMessage, that includes the needed attributes for the Connector as shown in this figure:

[tmberthold]

In the meantime, a presumably temporary workaround has been implemented in the EDC for Catalog-DescriptionRequests/Responses with large catalogs (Catalog with 5000 ContractOffers):

https://github.com/paullatzelsperger/DataSpaceConnector/blob/4320bbe922c422e2680dba3cf59c84a1fb239f5a/docs/developer/decision-records/2022-07-05/README.md

They use non-deterministic looping and send the CatalogRequest until the response no longer contains an offer.