Event List in Redis

Ideas for representing an event log in Redis satisfying

• We allow Redis to expire the list automatically;

• Keep the number of database round-trips as low as possible; and

• Poll request incudes a param indicating how much it has already seen, so that redundant replies are avoided.

The Ajax protocol goes something like this:

• Request is initially to a URL embedded in the page; this is opaque to the page nut happens to have the sequence number of the next event embedded in it.

• Response includes the following fields:

  • ready: true unless the events requested have been expired
  • next: URL # poll this URL next time (with updated sequence number)
  • pleaseWait: integer # number of milliseconds suggested delay before next call
  • pleaseReload: bool # events requested have already been expired
  • events: list of event # possibly empty

The response is encoded as JSON.

The list of events is in forward chronological order, which is the same as in sequence number order, and the first event has sequence number ≥ the param in the URL.

The Redis implementation is one persistent sequence key and one expiring list per board:

• kanbo:board-57-seq is the sequence number for the next event; and
• kanbo:board-57-events is a list of JSON-encoded events.

Years of db experience makes me chary of storing JSON in the database, but I think that is how Redis is supposed to work in this case.

When a request is made for sequence S we examine events[0] first and pull out its sequence number. If there is no such entry (because the list is expired or empty), get the sequence number from the seq key instead. If the sequence number is greater than S, we know events have been lost, so return {ready: false, pleaseReload: true}. Otherwise, we can calculate the range required and return that. With a little jiggery-pokery we can avoid having to JSON-decode all the events before re-encoding them in the response.

When adding an event, we use the value of seq and increment it, RPUSH to the list, and and set the expires value to 50 seconds.