Support change event

[gedw99]

It would be good if I can subscribe to a change feed at the golang api level.

This will allow any higher layer to be told when a table / document changes which makes building apps on top with subscription patterns

The event just needs to say what table / document and the nature of the change which is CUD ( create, update, delete )

This is amazingly useful to build apps on top of with whatever logic an app developer needs written by themselves.

I am not sure how hard this would be and it may be not possible for all engines.
If you have any thoughts on this then it would help me scope out the required work to do it.

Also has anyone tried running this using goMonile yet? Is it envisaged as a use case.

[asdine]

This would definitely be a great addition. I don't see it coming in the next release though because there is still some groundwork to be done.
In the meantime, if you need this feature right away, I think there is a "simple" way to do this at the engine level, by implementing the engine.Engine interface to wrap any engine and react whenever the Put or Delete method is called.

[gedw99]

so regarding simple way, you mean right after this line for example ?

https://github.com/genjidb/genji/blob/4a0b9c0b9ac6a152c35dad1103efa642f4792b5c/engine/badgerengine/store.go#L44

[asdine]

No, I mean by creating your own implementation wrapper. The idea is to implement the engine.Engine, engine.Transaction and engine.Store interfaces, while embedding another engine.

type myEngine struct {
  // embed any already existing engine
  // so you don't have to reimplement everything
  engine.Engine
}

// override only the Begin method
func (m myEngine) Begin(w writable bool) (Transaction, error) {
  tx, _ := m.Engine.Begin(w)
  return &myTransaction{tx}, nil
}

type myTransaction{
  // embed any already existing transaction
  // so you don't have to reimplement everything
  engine.Transaction
}

// override only the GetStore method
func (m myTransaction) GetStore(name []byte) (Store, error) {
  st, _ = m.Transaction.GetStore(name)
  return myStore{st}, nil
}

type myStore{
   // embed any already existing Store
  // so you don't have to reimplement everything
  engine.Store
}

// now override the Put and Delete methods


func (m myTransaction) Put(k, v []byte) error {
  err  := m.Store.Put()
  // run your trigger here
  return err
}

func (m myTransaction) Delete(k []byte) error {
  err  := m.Store.Delete()
  // run your trigger here
  return err
}

Then call it that way:

// create first the engine of your choice (bolt, badger or memory. Here I'll choose memory)
ng := memoryengine.NewEngine()

// then wrap it with yours
myNg := myEngine{ng}

// then open a database
db, err := genji.New(&myNg)

[gedw99]

Right i get it now. Thanks for explaining the best way to do it. Makes sense, and very doale.

I have another question that is related to this whole area of Change Event, and i think its worth visiting because its related to the way we do this.

Essentially, It would be nice if genji could be used for golang projects where you also need an embedded DB that has some HA / Replication properties. SO you can stand up 3 of them in 3 different regions for example.

One way to do the replication:
https://github.com/lni/dragonboat/tree/master/internal/logdb/kv/pebble

This is cool because we can get 2 things out of the one implementation.
a. Replication where you need it for Servers. When using genji in a Client App, you can just ignore this functionality.
b. Change events being exposed over some API to the golang Server App or Client App.

But not sure how you feel about this, and seeking advice....

[asdine]

There is already an issue about that here #48 I guess.
Basically I'm really interested but it's too early for us to focus on this aspect when there's not even basic support for LIKE or GROUP BY in Genji. I'd like to first make the core of Genji useful and stable, then focus on making it scale.

[gedw99]

Ok fair enough, i can see your point...

Is there an Issue for LIKE or GROUP BY ?

[asdine]

There is one for GROUP BY here #6, it is planned for the v0.8.0 milestone. LIKE will probably be done on v0.9.0 when we'll be working on regexp and string related features.

[gedw99]

Thanks for update on the 2 outstanding query aggregation function of "LIKE" and "GROUP BY". Really exciting !!!

In relation to change feed and my idea for doing it via a Raft like approach...
I have been playing with a Badger implementation that supports Clustering using Raft under the hood.
It also supports Change Feed via the Badger Subscribe function.

https://github.com/mkawserm/flamed

The standard Badger API is exposed by it. But i am still working out how best to use it with Genji.

example flags:

flamed run server --notify-commit true --node-id 1 --storage-path $(LIB_DATA_FSPATH)/data1 --http-server-address 0.0.0.0:8081 --raft-address 0.0.0.0:63001 --grpc-server-address 0.0.0.0:9091 --log-level debug

• "notify-commit" is the change feed functionality
• "raft-address" is the common RAFT IP:PORT
• It also exposes a HTTP and GRPC server for management function such as Auth.
• This can be run in standalone mode of clustered mode.

Under the hood it uses https://github.com/lni/dragonboat

• Uses Pebble ( https://github.com/cockroachdb/pebble) to do the RAFT aspects. It used to use RockDB, but pebble is now stable enough to be used for this.
• From what i can see its very fast

LICENSE looks liberal.

[asdine]

Adding this feature to Genji would be very exciting indeed 👍
I'll update this issue once the project will be ready to tackle it