feat(#9193): add functionality of getting people as pages or async iterables in cht-datasource

[sugat009]

Description

Added functionality of getting people as using pages or as AsyncGenerator using cht-datasource along with addition of REST endpoint /api/v1/person which provides people as pages.

Issues: #9237 , #9238 , #9241

Closes #9241

Usage:

1. Pagination API

Person.v1.getPageByType(Qualifier.byContactType(personType), cursor, limit); // limit and cursor are optional with defaults as 100 and "0" respectively

2. Async Generator API

const getAllIterator = Person.v1.getByType(ctx)(Qualifier.byContactType('person'));

3. REST API endpoint

HTTP GET api/v1/person

Query Params:

• personType - the person_type to fetch
• limit - default is 100
• cursor - default is 0

Code review checklist

• Readable: Concise, well named, follows the style guide, documented if necessary.
• Documented: Configuration and user documentation on cht-docs
• Tested: Unit and/or e2e where appropriate

Compose URLs

If Build CI hasn't passed, these may 404:

• Core
• CouchDB Single
• CouchDB Cluster

License

The software is provided under AGPL-3.0. Contributions to this project are accepted under the same license.

[sugat009]

The CI is failing for a param's JSDoc which exists but ESLint is marking it as does not exist.
https://github.com/medic/cht-core/actions/runs/10352638921/job/28653693985?pr=9311#step:11:88
Looking into it right now.
This might have been due to the upgrade of ESLint as this was passing before.

[jkuester]

This is looking great! Today I was able to make it through the cht-datasource implementation code (where I left some minor comments, mostly related to the null/undefined cursor discussion from Slack). Tomorrow I will continue with the rest!

[jkuester]

This is great! I have made it though the rest of the implementation code and all the tests. Just a number of nitpicks thoughout and a few logic tweaks, but otherwise this is nearly complete.

[jkuester]

Suggested change

	const responsePage = await getPage(Qualifier.byContactType(personType), cursor, limit);
	const responsePage = await getPage(Qualifier.byContactType(personType));

I don't want to get too deep into edge cases with these integration tests, but can we have this test cover the case where we use the default cursor/limit values. Then add another test where we call getPage twice. On the first call we give it a null cursor and a limit of 4. Then on the second call, we pass in the cursor value returned from the first call (and again a limit of 4). Then we can assert that between the two calls, we got the expected 7 persons back. One of the key things I want to be careful to assert in the new test is the value returned for the page cursor. Should be '4' on the first call and then null on the second.

[jkuester]

One more case that I thought of that we should probably have is to call the Person.v1.getAll function and load up all the returned persons and make sure we get all 7 of them. Just a safety check that our generator code integrates fine.

[sugat009]

I don't want to get too deep into edge cases with these integration tests, but can we have this test cover the case where we use the default cursor/limit values. Then add another test where we call getPage twice. On the first call we give it a null cursor and a limit of 4. Then on the second call, we pass in the cursor value returned from the first call (and again a limit of 4). Then we can assert that between the two calls, we got the expected 7 persons back. One of the key things I want to be careful to assert in the new test is the value returned for the page cursor. Should be '4' on the first call and then null on the second.

The reason I kept the integration test with the "non-paging" values for limit and cursor is that the objects being returned from the database are not in a consistent order for the order that we are passing into the utils.saveDocs() function. In our case, we pass const allDocItems = [contact0, contact1, contact2, place0, place1, place2, patient]; into utils.saveDocs but it is not guaranteed that contacts will be returned in the same order. The order is undeterministic and random; it could be the order in which they were indexed into the view, so the page cannot be compared. For example, if I request a page with 4 items, it could be any 4 items.

[jkuester]

The order is undeterministic and random; it could be the order in which they were indexed into the view

Minor, but I just want to clarify that order the data is returned should not be random. My understanding is that data returned from couch views is deterministically ordered based on the key value and in cases like this where we have many results for the same key, the order is by _id value. Each call to getPage with the same parameters, should return the same set of data (otherwise our skip-based paging would not work 😓). However, this does not help us much for these integration tests since, as you have observed, the UUID's assigned as the _id value for our new contacts are random and so, the order of the returned data could change between test runs.

All that being said, I don't think we need to worry too much about trying to assert the order of the data returned. We don't even need to assert the exact docs returned for each page. For the test where we get multiple pages, it seems enough to just assert the size of the data array for each page. Then at the end combine the data arrays into one array that contains all the contacts and do your equalInAnyOrder assertion to make sure that ultimately we got back all the contacts we expected.

[jkuester]

Left a couple other minor comments around the tests, but otherwise this is ready to go!

One thing we do not want to forget, @sugat009, is that we need to update the api reference docs to include documentation for the new REST endpoint. Here is one of the previous PRs for example.

[jkuester]