Orderbook Pagination

[kodemon]

Following our recent discussion on potential approaches to introducing pagination in the order book system in anticipation of a large number of orders, it is crucial to develop a comprehensive plan of action to effectively address this issue.

Current Workflow

At present, when a client initiates an order book request, the sado-api forwards the address for the order book to the ordit-api, which retrieves a list of transactions. This list encompasses all the transactions present in the order book.

Issue

The core concept of pagination involves presenting a subset of a larger dataset to optimize data delivery and minimize payload sizes. However, the current structure necessitates pagination implementation at the ordit-api level. This presents a challenge as the sado-api compiles its results holistically, requiring the entire dataset to determine the state of orders and offers.

Consequently, it is impractical to implement pagination on the ordit-api.

Proposed Solution

Given that the sado-api demands a comprehensive view of the order book's state within the network, all relevant data must be accessible at the sado-api level. As such, pagination, sorting, and filtering should be executed within the sado-api layer without relying on the ordit-api for paginated data.

We propose the establishment of an order book database within the sado-api, which will be updated based on changes in the ordit-api. By managing the order book through a data layer in the sado-api, the complete order book state can be accurately represented, allowing for pagination, filtering, and sorting without losing essential data points required to present an accurate state to the client.

To accomplish this, the connection between the sado-api and ordit-api will serve as a hydration point where either the ordit-api feeds updates to the sado-api, or the sado-api subscribes to or polls the ordit-api periodically for updates. Subsequently, the latest changes will be incorporated into the sado-api database, which will, in turn, update the result set provided to the client. This approach may also prove beneficial for future websocket subscriptions from the client to the sado-api, as it enables the sado-api to manage all logical layers comprehensively.

Timeline

Since this is not a trivial task to resolve the timeline for this functionality will potentially require an optimistic 1 - 2 week timeline with major changes being made in sado-api and minor feature additions to ordit-api.

As such we may want to prioritize other must have tasks for demonstration and/or release before tackling this problem head on.

[kodemon]

I have actively been working on prototyping a solution for this proposal and I am about 40% done with a solution I think can work. My focus right now is to just keep that momentum going.

Will detail more in our discussion later today.