Make freetext search view indexes "offline only"

[jkuester]

Is your feature request related to a problem? Please describe.
If we move to Nouveau for freetext searching on the Couch server (aka "online" use), then we still need a solution for freetext searching "offline" (what offline users would experience via their local Pouch instance).

Describe the solution you'd like
The most straightforward approach should be to just have a design doc that is "client-only" similar to how we have docs that are currently "server-only". This design doc will be indexed by PouchDB clients, but will not be indexed by the Couch server.

Since we are likely going to require a complete re-indexing of these views on the client devices one way or the other, we should take this opportunity to remove the unused key:value emissions from the freetext views as originally proposed for the "freetext-lite" views.

Describe alternatives you've considered
One interesting alternative would come after our upgrade to Pouch 9 and the indexeddb adapter. IIRC, Mango queries should be more performance on Pouch with indexeddb. If that is the case, then we could evaluate possibly using Mango queries for freetext searching. This one blog post suggests it works well... 😅

[jkuester]

Here are my conclusions after taking a deep dive into possible implementations. The main approach options seem to be:

1. Add ddoc to Couch as normal but with "autoupdate": false so it does not get indexed on the server. Then in webapp we would need to warm the index when the app starts.
   • Pros:
     • Design doc is managed/controlled along with the rest. Gets all the metadata, etc.
   • Cons:
     • Run the risk of someone calling the view (even from Fauxton) and then it will be built on the server-side.
     • Have to manually warm the view on the client side (since it will not auto-update).
     • Need a bit of additional logic to allow replication of the new ddoc to the clients.
2. Add ddoc directly on the client side. Hardcode the ddoc directly in the webapp code and have it build the index on the client side.
   • Pros:
     • With the ddoc only on the client, there is no danger of it getting accidentally indexed on the server.
     • Might also simplify logic in shared-libs/search since that code could just check if the offline ddoc exists. If it does, use it, otherwise call nouveau.
   • Cons:
     • New code flow - not tracking ddoc with rest of the server-side ones. View code will not all be in the same place in the repo.
     • Need custom logic in webapp to make sure the index is up-to-date when the app starts.
3. A middle alternative would be to add logic to our server-side handling of ddocs that would allow for the creation of a new doc on the Couch server with the contents of a ddoc, but without the _design/* id. On the server it would just be a normal doc (with a new type). But, when it gets replicated to a client, the doc gets transformed into an actual ddoc (either in the webapp or by the server's replication logic).
   • Pros:
     • Get benefits of #1 without the drawbacks
     • Still would allow for simple shared-.libs/search code
   • Cons:
     • Requires additional ddoc logic on the server side for the creation/management of the new type of docs.
     • Server-based transformation into an actual ddoc could be tricky/impossible. Seems unlikely that Couch/Pouch wants you changing doc ids mid-replication....
     • If we just do client-side transformation, it is unclear what benefits this offers over #2.

Alternative rejected:

I also took some time to look into any Pouch-specific alternative approaches to freetext searching (since we will no longer be tied to solutions that also work in Couch). Unfortunately, there is absolutely nothing viable here.

My original hope was that Mango text indexes would help, but the pouchdb-find plugin only supports json indexes and none of the other Mango features are going to be helpful doing incremental searching over all fields in a doc. In fact, though the Couch docs mention the text index type, they also vaguely note "Optional Text indexes are supported via a third-party library". I suspect this is actually a Cloudant-specific feature....

Beyond Mango, any discussions of freetext searching in Pouch seem to mention pouchdb-quick-search which has not been updated in 7 years. To make matters worse, pouchdb-quick-search is based on lunr.js which has not been updated in 4 years. So, any move in that direction seems like it would basically require a re-write of the plugin....

[jkuester]

Also here is my conclusion and next steps based on the above analysis:

#1 just seems too risky given that someone clicking through Fauxton could trigger the index to build. #3 seems unnecessarily complex for dubious benefits. While it would be nice keep the offline index code as close as possible to the rest of the indexes, it does not seem to justify the amount of moving parts that this approach entails.

So, I am currently doing further investigation and prototyping for #2.

[dianabarsan]

I also agree that #2 is simplest. And probably more correct overall.

Right now, when the app gets an update, there are two things that need to be updated: 1. the medic-client ddoc and 2. the actual app code. We do tell people that they should reload their app to use the new code, but it's entirely possible some app somewhere runs and old webapp code with the new ddoc.

I remember we even tracked this through telemetry on some instance that was reporting errors at some point.

There's an issue: #7146

[garethbowen]

Have to manually warm the view on the client side (since it will not auto-update).

As far as I know pouchdb doesn't do background indexing, and ignores the autoupdate setting, and we don't "warm" it anyway so this isn't a "con", just business as usual for client side.

Another option would be to store the client only ddoc as an attachment on the medic-client ddoc, which the client syncs and extracts on change. A "pro" of this is that then the ddocs are all synced together, but a "con" is weird errors if the change event doesn't fire or gets interrupted. You would probably still need to check on startup to ensure the client only ddoc is up to date, which makes it very similar to number 2.

I think I'd go for #2. Checking whether it's up to date is a pain but I guess hash it at build time and compare the hash on first start.

[jkuester]

I have done some more research here both into the Nouveau api and into the existing functionality of shared-libs/search. TLDR is that Nouveau should be able to integrate fine with our paging pattern in cht-datasource:

By default, a nouveau index response returns 25 rows. The number of hits that are returned can be changed by using the limit parameter. Each response includes a bookmark field. You can include the value of the bookmark field in subsequent queries to fetch results from deeper in the result set.

My current proposal is that we implement the freetext searching for contacts/reports in cht-datasource. Then, we update shared-libs/search to call cht-datasource. This should be possible with minimal complexity because shared-libs/search has basically two usage modes with regards to the view indexes:

• Single view mode: when only one view index is being used, shared-libs/search will transparently pass through the skip/limit params from the consumer. This will be trivial to support in cht-datasource with our getPage function pattern.
• Multi-view mode: when a search involves multiple view indexes, shared-libs/search will fetch all results from each view index (querying the views without a limit). Then, if the consumer provided skip/limit params, the shared-libs/search logic will do some manual "paging" of the results that it compiled. As Diana noted, this is a mind-numbingly inefficient process, particularly on the server. However, there is no clear alternative solution. Regardless, it should still be pretty easy to support this flow by returning freetext search data from cht-datasource via the getAll pattern and just streaming all the available results.

I do not think we should implement multi-view searches in cht-datasource (unless we can come up with a feasible way to do paging). Instead we can start by implementing just the single-view freetext searches in cht-datasource for now. Later we can add support for the other single-view searches. (Ultimately, I think it would be best to refactor any code that does not need multi-view searches to just call cht-datasource directly, instead of calling through shared-libs/search. This would make it much easier to understand how/when multi-view searches are happening and hopefully eventually remove them or replace them with dedicated views (this might be more feasible with Nouveau).

@m5r @sugat009 I would love your feedback on this proposed cht-datasource API. Thanks!:

// ContactTypeQualifier already exists
type FreetextQualifier = Readonly<{ freetext: string }>;

// New REST api: /api/v1/contact/id
namespace Contact {
  const getIdsPage = (context: DataContext) => (qualifier: ContactTypeQualifier | FreetextQualifier, cursor: Nullable<string>, limit: number) => Promise<Page<string>>
  const getIdsAll = (context: DataContext) => (qualifier: ContactTypeQualifier | FreetextQualifier) => AsyncGenerator<string, null>
}

// New REST api: /api/v1/report/id
namespace Report {
  const getIdsPage = (context: DataContext) => (qualifier: FreetextQualifier, cursor: Nullable<string>, limit: number) => Promise<Page<string>>
  const getIdsAll = (context: DataContext) => (qualifier: FreetextQualifier) => AsyncGenerator<string, null>
}

[jkuester]

One additional design note with regards to the cht-datasource implementation. We are still going to need a check in the local adapter code to see if the new client-side index exists. Basically here is the logic breakdown:

• Remote Adapter:
  • Calls REST endpoint (backed by cht-datasource in api server) - /api/v1/contact/id?type='person'&freetext="hello world"
• Local Adapter:
  • Checks if _design/medic-offline-freetext exists
    • If yes, then queries that view index - this is the case for offline users in webapp
    • Else, queries nouveau endpoint - this is the case for api/sentinel

[garethbowen]

Checks if _design/medic-offline-freetext exists

This step raised a red flag for me because it feels unnecessarily complex. I'm guessing this is forced from the pattern of using cht-datasource on both the server and client and trying to reuse code which is valiant. In other code the views you call on the client and the server are the same so it makes sense to share the code. But I think this is a special case, because here there is no shared code - the call to the Nouveau server will only ever be made on the server. Therefore I think the implementation of the REST API should be custom server code and not part of cht-datasource. I think this will make it easier to follow what's going on in the datasource code.

Also a question about the index itself - I think we should be aiming for a single Nouveau index for all contacts and reports with one of the indexed fields being "type". My gut feeling is this will be more efficient (disk, memory, build speed) but that could be completely wrong. One way it would almost certainly be worse is if we change the way we index reports it will have to reindex all contacts as well even though nothing has changed, however it's quite likely that lucene reindexes so much more quickly that we shouldn't worry too much about that. If we go with the "one index" approach then it would make sense for the REST API to be more generic so you could theoretically search for either reports or contacts, eg: /api/v1/search/?type=ABC&patient_id=XYZ

[sugat009]

@jkuester On the API design. Let's be more explicit with the endpoint naming. If only one endpoint as Gareth suggested is opted then, /api/v1/search else /api/v1/contact/search and /api/v1/report/search with the internal methods as searchPage and searchAll. (Even though getting is essentially searching)

[jkuester]

I think the implementation of the REST API should be custom server code and not part of cht-datasource.

I 100% agree that this is a bit smelly since the Nouveau code is only used on the server side (and the offline-only views will only be used on the client side). The problem, though, is with regards to the multi-view searches. The api export endpoint basically gives full access to the shared-libs/search api, allowing consumers to construct ad-hoc multi-view searches (e.g. search for reports by freetext and by form type). The logic for multi-view searching currently lives in shared-libs/search and so, if we do not have the Nouveau logic in cht-datasource, we would still need to put it in shared-libs/search (so it knows which search to use for the multi-view queries). Then, shared-libs/search is still going to need some way to figure out if it should use Nouveau or cht-datasource.

I just have not been able to come up with any approach yet that would be cleaner then encapsulating the Nouveau logic (and the view check) in cht-datasource and letting everything else just flow through.... The ideal scenario, IMHO, would be to make a breaking change to API (CHT 5.0?) where consumers are no longer allowed to do multi-view searches via the export api. Then we could refactor the API code to not use shared-libs/search at all. Then the api could do all its freetext searching directly with Nouveau and cht-datasource would not need to have that logic.

IDK... maybe it makes the most sense to have it in cht-datasource for now, but remove it later if we update the export api? Alternatively, we could keep the cht-datasource logic clean by pushing the Nouveau check up into shared-libs/search. The search parameters for that library are already "flexible", so we could have api pass some flag to indicate that Nouveau should be used. Layer the jankyness on top of the jankyness instead of polluting cht-datasource....? 😅

If we go with the "one index" approach then it would make sense for the REST API to be more generic so you could theoretically search for either reports or contacts

On the API design. Let's be more explicit with the endpoint naming

Yeah, I think we have a lot of unanswered questions currently about how the Nouveau index(es) should be configured. That is all more of an investigation for #9542, though it could definitely influence how we structure the REST/cht-datasource apis.

I do want to push back a bit on the idea of having a /api/v1/search or /api/v1/contact/search endpoint. My understanding of REST "best practices" (I put that in quotes because I do not claim to be a REST expert, so that should just be read as: "stuff that I have read on the internet"... 😅) is that your endpoints should represent resources and your HTTP methods should be actions that you can take with those resources. That is where I got the /api/v1/contact/id endpoint as representing contact identifiers. Then doing a GET against that endpoint makes sense semantically that I am trying to retrieve contact ids (filtered by query parameters that I might provide). It seems like the alternative of having the endpoint represent an action (e.g. /api/v1/search) is less "RESTful".

Based on our current search implementation in shared-libs/search, we have not needed to implement any kind of generic search functionality that gets results containing both reports and contacts. However, if we ever decided we needed this, it seems we could have some endpoint like: /api/v1/doc/id or /api/v1/entity/id or something generic like that....

[garethbowen]

The api export endpoint basically gives full access to the shared-libs/search api, allowing consumers to construct ad-hoc multi-view searches (e.g. search for reports by freetext and by form type).

Ahh I see the problem. Thanks for walking me through it!

There is potentially another solution if we're willing to double down on Nouveau to replace the entire server side search with the lucene search by indexing all the fields we can currently filter on. This makes the shared-lib/search implementation completely different between server and client. This would solve the "mind-numbingly inefficient process" the breaks the server, but is a massive gamble on the as-yet unproven Nouveau, so let's stick to the MVP as you've described it, and we can replace the jank with the full Nouveau experience later without a break change.

I do want to push back a bit on the idea of having a /api/v1/search or /api/v1/contact/search endpoint.
However, if we ever decided we needed this, it seems we could have some endpoint like: /api/v1/doc/id or /api/v1/entity/id or something generic like that....

Your naming is better so let's stick with the convention of resources not actions. I agree with your pragmatic approach of providing the specific APIs first, and adding the generic one later if and when needed.

From my understanding lucene works best when dealing with one big heap of docs and loads of keys, rather than separate index silos. This also gives us a lot of flexibility to write future server side queries without having to reindex. So even if we don't expose it via an endpoint just yet I'd like to build a little future proofing into the index. Caveat: these assertions need to be tested for our use case!

[jkuester]

FTR, I have split the cht-datasource changes off into their own sub-issue: #9586

This sub-issue will just be focused on adding offline views to webapp (and a bit of logic to actually use those views in shared-libs/search which will get copied to cht-datasource in the other issue).

[jkuester]

As originally discussed on Slack, there is a complication with removing the key:value indexes even in webapp. 😞 The shared-libs/validation package does key:value queries against the reports_by_freetext index. These validations get run during several Sentinel transitions:

• accept_case_reports
• accept_patient_reports
• muting
• registration
• update_notifications

The relevant one for this case is muting because the webapp also supports offline client-side muting. To maintain consistent behavior with the server-side muting, the webapp client-side muting transition executes the same validations for the mute/unmute reports. To be clear, the reports_by_freetext query is made by shared-libs/validation when an exists rule is configured, validating that another report does not already exist with the same field value.

The simple conclusion here is that we have to keep the key:value indexes even in the offline freetext views. However @garethbowen I would be interested to hear your thoughts on this because of these additional factors:

• The exists rule docs state:

  • As of 2.12 most uses of this function are obsolete because checking for a valid patient_id is done automatically by the accept_patient_report transition using registration_not_found in the messages.event_type.

  • Perhaps there is reason to expect that this rule is no longer in use?
• More notably, the exists queries triggered by the webapp's client-side muting validation only run against the docs accessible to the user.
  • This means the functionality here is already "flawed" in that it does not match the behavior of the server-side muting transition (which will run the validation exists query against all docs in the db).

IDK, I still cannot come up with any way to avoid the key:value indexes in our offline views that does not involve a technically breaking change to the muting validation logic, but it seems a shame to lose our opportunity to reduce the on-device view size without at least getting a second opinion here.... 😅

[garethbowen]

@jkuester I can't think of a good reason to run validation during muting on the client side. Firstly these validations are rare for xforms because it's better handled through xform validations. Secondly as you say it's flawed because if validation passes on client side that's no guarantee it'll pass on the server with all docs available.

The validation like something that was added by accident when we added muting to the client and was no intentional. So in general I would seriously look at removing it in order to improve client side performance by reducing the size of the views.

However the original driver for this effort is server side TCO so this is out of scope. Assuming it's straightforward to keep the existing views in the current state let's just do that and raise an issue to optimise client side views at a later date.

[jkuester]

However the original driver for this effort is server side TCO so this is out of scope. Assuming it's straightforward to keep the existing views in the current state let's just do that and raise an issue to optimize client side views at a later date.

@garethbowen Agreed! The only extra consideration is that I have not found any way to move the current freetext views from one design doc to another (e.g. to the new offline doc that only exists on the clients) that does not require a full rebuilding of the views. This means that we are going to trigger a full rebuild of these indexes once when switching to the offline index and then again when we go do to this key:value cleanup.

It is a bit tricky to profile the exact performance implications this will have since it is highly dependent on the device specs and the number of docs associated with the user. To just give us some reference numbers, I created a test user with ~6000 reports (plus ~600 contacts). I logged in as the user and synced all the docs. Then I tried running a search on the Reports tab. Since we do not warm the views in the webapp, the view is built at the time of the first search. With my laptop browser running with full power it still took ~45sec for the search to complete. If I throttled Chromium to have "Low-end mobile" settings (6x CPU throttling) then it took 6.5min to complete. This is just for building the medic-client/reports_by_freetext view when all the docs have already been replicated. (For the record, I also verified that the behavior/timing is basically the same if I make an update to the medic-client/reports_by_freetext view after it was initially indexed. The first search after re-loading the app triggers the rebuilding.)

These numbers are not the end of the world, but they will be incurred by every device for all offline users. But, this technically is the same price that is always paid when logging in as an existing user on a new device, so maybe I should not worry so much... 🤷

[jkuester]

I have not found any way to move the current freetext views from one design doc to another

After further investigation, this might not be true! I did a deep dive into the Pouch caching functionality and it appears the cache keys are only based on the contents of the map/reduce functions. So, view indexes with the same functions should hit the same cache?!? I have to do more testing here to confirm. Specifically, I am still a bit unsure of the viewCleanup logic. It seems to allow for the possibility that multiple views dbs could reference the same view cache, but I will have to test more to confirm.

If this is true, though, then it gives us a clear benefit to just copying the existing freetext views into our offline design doc (and avoiding forcing a re-indexing). 👍

[1yuv]

Hi @garethbowen , @jkuester , Related to this, SMS workflows use different CHT validation rules directly to do various validations that query freetext search views. These rules use query exists and other functions which queries views like reports_by_freetext and others. Since SMS users do not log in to any specific devices, they won't have any offline storage and we'd still need these freetext views to be there on the server.

[m5r]

@1yuv the plan is to keep a way to search through reports and contacts on the server, but to do it in a more storage-effective way with CouchDB Nouveau. The effort to research this new search implementation is tracked in #9542.

[jkuester]

These rules use query exists

@1yuv, out of curiosity, what are the validations doing with the exists function? I don't think exists is formally deprecated, but I thought usage might be minimal after reading in the docs:

As of 2.12 most uses of this function are obsolete because checking for a valid patient_id is done automatically by the accept_patient_report transition using registration_not_found in the messages.event_type.

These freetext searches are a very blunt tool and it would be good to know if there are other specific use-cases that could be properly optimized with dedicated indexes.

[1yuv]

Hi @jkuester , we use some functions like unique which uses exists in cht-core. We use unique to prevent duplicate pregnancy registration. We are planning to use exists directly after this issue is released on 4.15. The use of exists will be to ensure that pregnancy is registered before someone sends delivery form for patient.

[jkuester]

Okay, so the final conclusion here regarding if it is possible to "move" a view to an offline-only ddoc without triggering a re-index is, NO, it is not possible. 😞 As detailed in pouchdb/pouchdb#9006, the new offline index would share the cached view index from the original medic-client view, but the cache will be automatically deleted when the original medic-client view is removed.

@garethbowen this brings us back to the question of if we want to just pay this re-indexing cost twice (once to make the index offline and again later to remove the key:value indexes) or if we should just remove the key:value indexes now (along with support for report validation in the offline muting workflow)? One final "option" here is that I believe it might be technically be possible for us to manually hack the _local/mrviews doc to point to our new offline design/views and avoid the cache value being cleared during viewCleanup. It would be sketchy.

[garethbowen]

I was wondering about the manual view move hack too but it seems too risky.

It feels like there's another option where instead of having two different client side ddocs with different names, what about one ddoc with different views? I think it's only slightly more confusing than the other option and means we can keep the views.

Another option, which is potentially a complete diversion, is to switch to pouchdb 9 and use the idbnext adapter which builds views in a completely different way, may be much faster, and at least there's a gain to come from the forced client side rebuild. We're risking turning this feature into a giant ball of improvements but it might make it palatable.

Otherwise, yes, if we're paying the view build cost let's re-re-review the index definition to be optimal (ie: remove unused keys).

[jkuester]

instead of having two different client side ddocs with different names, what about one ddoc with different views?

If by "different views" you mean "the same views", then this might work. Ultimately, the plan is to remove the contacts_by_freetext view from the medic-client ddoc so that this view is not built on the server. If there is no view with the name contacts_by_freetext in the medic-client ddoc on Pouch when viewCleanup is called, then Pouch will delete the cache for this view. In theory, we should be able, in the webapp code, to re-insert the contacts_by_freetext onto the medic-client ddoc (instead of creating a new offline ddoc as originally planned). If this happens before viewCleanup is called, I think it should work.... 🤔

Another option, which is potentially a complete diversion, is to switch to pouchdb 9 and use the idbnext adapter which builds views in a completely different way,

Ohh the temptation! This is a good point about the full re-index coming from the switch to idbnext. We could do all three at once and only pay the re-indexing cost once! (Especially since these freetext views represent the majority of the client-side indexes...)

How do you feel about the long-term maintenance of offline-only views being inserted into the medic-client ddoc instead of just having a separate dedicated offline-only ddoc being created by webapp? Initially I was thinking that modifying medic-client was pretty sketchy, but the more I think about this, the less of a difference I see between these two options... My thought here is that if we don't like the idea of having the offline views in medic-client long-term, then we should just bite the bullet now and try to ship everything at the same time (using a new dedicated offline ddoc). On the other hand, if editing medic-client seems pretty maintainable, then perhaps it is best to go that approach and avoid the index rebuild with these changes...

[garethbowen]

How do you feel about the long-term maintenance of offline-only views being inserted into the medic-client ddoc instead of just having a separate dedicated offline-only ddoc being created by webapp? Initially I was thinking that modifying medic-client was pretty sketchy ...

Oh it's definitely sketchy. The main issue I see is IIRC we block modifying the ddocs on the client so you'd have to change that, and then find a way to stop them trying to replicate back to the server. This is similar to having a separate ddoc that you don't want to replicate, but it's just a little sketchier because getting this wrong would be more destructive than having separate ddocs.

[jkuester]

I did some quick prototyping in the browser and I am able to modify the ddoc just fine (and add a view view). And, we are already blocking the ddocs from replicating back to the server, so that should not be an issue.

However, I think I found the deal-breaker and that is the fact that when we modify the ddoc on the client but do not sync the modification back to the server, things get unpredictable for what happens when future updates to that ddoc are replicated from the server to the client. In my testing, if I only made one change to the ddoc on the server, then Pouch on the client would choose the local version of the ddoc as the winner and it would not reflect the server-side changes. OTOH, if I made multiple revisions to the ddoc on the server and then tried to replicate them all at once, Pouch on the client would choose the server version as the winner and replace my local changes with the server version. This kind of thing is a hard pass for me and I think we should instead move forwards with the plan to stick to a separate offline ddoc and try to align these changes with idbnext and removing support for validations when muting offline.

[jkuester]

I have logged #9652 to cover the removal of validations during the client-side muting transition.

If we move forwards with that issue, it should be completed before this issue here (so that we can safely not include the key:value entries in the new offline indexes).

Either way, we want to try to align this issue with #9208 (switching to idbnext) so we one have one big re-indexing effort.

[dianabarsan]

Possibly related: #9003

[dianabarsan]

Either way, we want to try to align this issue with #9208 (switching to idbnext) so we one have one big re-indexing effort.

I believe #9208 is a long way away, and we could aim to deliver the TCO improvement sooner.