Filtered replication

[hongnk]

Using a couchdb backend, socket pouch doesn't read filter set in replication options and therefore emit all changes to all connected clients.

[nolanlawson]

Can you provide a test to reproduce? socket-pouch runs the entire PouchDB test suite and there is definitely a filter test in there.

[hongnk]

To be exact, the changes do not replicate (perhaps they are filtered out at client side pouchdb). However I observe that data are still sent over the wire when looking at websocket frames in chrome debugger. This is different from couchdb HTTP long poll, since it doesn't emit out any data that don't match filter condition set by client.

This is an undesired effect since unnecessary data are sent over to non-relevant clients.

In my test case, 2 or more clients are listening for changes using different channels. However as mentioned, messages meant for channel A are sent to channel B as well.

I'm still not familiar with pouchdb test suite, but my test code is as follows:

var socketpouch = new PouchDB({
    adapter: 'socket', 
    name: 'test', 
    url: 'ws://localhost:8000' (this socketpouch will proxy a couchdb at the backend)
})

//add a design doc to couchdb, to allow filter by channel
socketpouch.put{
  "_id": "_design/bychannel",
  "filters": {
    "channelfilter": "function(doc, req){if (doc.channel==req.query.channel){return true;} else {return false;}}"
  }
}

//client will listen to particular channel, in this case channel A
var pouchdb = new PouchDB('localpouch')
pouchdb.sync(socketpouch, {
    filter: "bychannel/channelfilter",
    query_params: {channel: 'A'},
    live: true
})

//add doc or update doc, this should replicate to other clients that filter for channel A
pouchdb.put({
    doc_content: 'xyz',
    channel: 'A'
})

[faraazc]

@hongnk hey i think you are missing something. you need to add design document in the localdb as well.follow below link
http://pouchdb.com/2015/04/05/filtered-replication.html

and let me know if that works?

This is critical bug i think if that does not work.

please update as soon as possible

[nolanlawson]

In my test case, 2 or more clients are listening for changes using different channels. However as mentioned, messages meant for channel A are sent to channel B as well.

This sounds like a separate bug. Can you confirm that the two clients have the same "messageId" and that's why they're getting the same messages? If two clients are getting the same messages, then that's a security issue and very different from a performance problem.

you need to add design document in the localdb as well.follow below link

Yep, that's true. If you use an ad-hoc filter function, then the filtering will be done client side. However it seems like @hongnk is not doing that.

I'm still not familiar with pouchdb test suite

Here's how you can debug this: read the README guide to testing, which will set up the tests in a browser. Navigate to the test in a browser, add ?grep=filter to the URL bar, which will search for any tests with the word "filter" in them. See what happens for tests that use design-doc-related filter functions.

It's possible that the docs are unnecessarily being sent over the wire, but from what I recall of how I implemented this, that is not possible.

[hongnk]

Thanks nolanlawson for looking at this.
@faraazc yes the design doc is added in local pouch, I just didn't mention the previous message.
@nolanlawson I don't think messageId is the issue, because one client on Chrome and the other on Firefox.

I put all these codes in a working sample and attach here so you can test it out. It's a zip file but I rename as pptx to be able to upload here.
test.pptx

Reproduction steps:

1. Need pouchdb-server or couchdb, node and socketpouch (can use the sample socketpouchserver.js included in zip file)
2. On Chrome, go to: http://localhost/testfilterbug.html?channel=A
3. On Firefox, go to: http://localhost/testfilterbug.html?channel=A
4. Click "generate doc" button on Firefox, and watch Chrome websocket debug frames. the generated doc is synced to Chrome as expected because they filter for same channel A. Note: all generated doc IDs are prefixed with channel A or B depending on querystring, so it is easier to watch.
5. On Firefox, switch to: http://localhost/testfilterbug.html?channel=B, and watch Chrome websocket log, the messages are still coming, while I expect that they should not be sent over to Chrome since it filters for channel A only.
6. You might try to switch to direct HTTP to couch server instead (I have a commented line in the html file to do that). And watch XHR message, it works as epxected, message are sent only if Firefox is on channel A, but not when switched to channel B.

[faraazc]

@nolanlawson when shall we expect this to be fixed?.

[nolanlawson]

@faraazc I am not currently putting socket-pouch high on my priority list, because it's a project I did for fun and am not currently using myself. So the answer is: never, unless somebody else takes it up.

[pradiprv]

@nolanlawson filter parameter is not sent to websoket message that's why filtered replication is not working. Add socket adapter in if condition in pouchdb library to fix this bug.

See code

Current:

if (this.db.type() !== 'http' && !opts.doc_ids) {
      return this.filterChanges(opts);
}

Should be:

if (this.db.type() !== 'http' && this.db.type() !== 'socket' && !opts.doc_ids) {
      return this.filterChanges(opts);
}

[hongnk]

@pradiprv thank you the issue is confirmed fixed after adding if condition. Currently I also follow @nolanlawson suggestion, after instantiate socketpouchdb, overriding it with socketpouchdb.type() = function() { return "http"; } as a work-around.

[TomKaltz]

I tried following the related issues and can't figure out if this is fixed. Can you someone fill me in?

[pradiprv]

@TomKaltz I have forked pouchdb and socket-pouch to fix this issue. Add following as bower dependency.

"pouchdb": "DreamworldSolutions/pouchdb#^6.2.0"
"socket-pouch": "DreamworldSolutions/socket-pouch#^2.0.2"

[vshl92]

The implementation has changed in current version 8.0.1, so above provided solution didn't work for me. However i could still find a solution.

Issue:

Facing 403 forbidden error on db replication over web sockets with a proxy server using library "socket-pouchdb".

Solution:

I could solve the issue just by setting "pouchdbinstance._remote = true".

Explanation:

While using web socket adapter "pouchdbinstance._remote" is not set to true.

See code :

• https://github.com/pouchdb/pouchdb/blob/7d71f444e61cc6b4d5da14998a959bb62a0bfb09/packages/node_modules/pouchdb-utils/src/isRemote.js#L15
• https://github.com/pouchdb/pouchdb/blob/7d71f444e61cc6b4d5da14998a959bb62a0bfb09/packages/node_modules/pouchdb-core/src/changes.js#L175
• https://github.com/pouchdb/pouchdb/blob/7d71f444e61cc6b4d5da14998a959bb62a0bfb09/packages/node_modules/pouchdb-changes-filter/src/index.js#L47

For "pouchdbinstance._remote = false/undefined", it makes a get call for doc_id = "_design_id/filter_name", which fails with 403 forbidden error. By setting db._remote = true, it does not make this get call and avoids 403 forbidden error.

Implementation:

const remotedb = new PouchDB({
  adapter: "socket",
  name: "<db_name>",
  url: "ws://localhost:80", // your ws proxy server url
});
// Note: for http adapter, _remote is by default true
// For socket adapter, set  _remote true explicitly 
remotedb._remote = true;