Messages are lost after a large amount of input

[NerdyBoyCool]

Thanks for developing a great gem.
And thanks for answering some of my previous questions.

We have been using this gem for the past few months as a beta version and have found cases where messages are lost under certain conditions.

First, some information about our environment:
Ruby on Rails: 7.0.4
Ruby: 3.2.1
Backend Data Source: Redis
Frontend Frame Work: Next.js (We use y-actioncable npm package in app.)

Our scripts. For debug, I put some loggers.

class DocumentChannel < ApplicationCable::Channel
  include Y::Actioncable::Sync

  def subscribed
    reject unless document

    sync_for(document) do |id, update|
      redis.save(id, update)
    end
  end

  def unsubscribed
    stop_stream_from canonical_channel_key
  end

  def receive(message)
    sync_to(document, message)
  end

  def doc
    @doc ||= load do |id|
      redis.load(id)
    end
  end

  def document
    @document ||= organization.documents.find(params[:document_id])
  end

  def redis
    @redis ||= Yrb::Editor::Redis.new
  end
end

module Yrb
  module Editor
    class Redis
      attr_reader :client, :logger

      def initialize
        @client ||= ::Redis.new(url: Rails.application.config.x.redis_url)
        @logger = SemanticLogger[self.class.to_s]
      end

      def save(id, update)
        begin
          data = encode(update)
          doc = client.set(id, data)
          logger.info("Yrb::Editor::Redis Finish Save", name: self.class.to_s, id:)
          doc
        rescue ::Redis::BaseError => e
          logger.error("Yrb::Editor::Redis Failed to Save", name: self.class.to_s, inspect: e.inspect, message: e.message)
        end
      end

      def load(id)
        begin
          data = client.get(id)
          if data.present?
            logger.info("Yrb::Editor::Redis Finish Load", name: self.class.to_s, id:)
            decode(data) unless data.nil?
          else
            logger.error("Yrb::Editor::Redis Not Found", name: self.class.to_s, id:)
            nil
          end
        rescue ::Redis::BaseError => e
          logger.error("Yrb::Editor::Redis Failed to Load", name: self.class.to_s, inspect: e.inspect, message: e.message)
        end
      end

      def encode(update)
        update.pack("C*")
      end

      def decode(data)
        data.unpack("C*") unless data.nil?
      end
    end
  end
end

---

Kapture.2023-09-04.at.15.32.41.mp4

In the video above, the message is saved correctly for normal input, but after a large amount of input, it is lost after the page is reloaded. (Lines of text beginning with "e" are missing.)

Could #25 be related to this issue?

If you want any information to understand the issue, please let us know.
Thank for reading.

[NerdyBoyCool]

@eliias
I would be glad to get your opinion on this issue.

[eliias]

Hm… can't tell from the video. It would be good to log incoming messages on the server? There are two possible places where this could get lost, before transmitting from the client to server, and before storing to e.g., Redis on the server. I have definitely worked with bigger chunks of text than this before.

[eliias]

#25 is a more holistic approach to delivery guarantees that would let the client know when a message is successfully distributed/persisted. A client would therefore store all deltas locally up until the point where the server acknowledges the retrieval of the update. This could be used to e.g., store local changes to LocalStorage or other persistence mechanisms, and the client would retry the broadcast up until its successful.

[PragmaFlow]

Has any more investigation into this been done? I am experiencing the same issues.

[eliias]

Has any more investigation into this been done? I am experiencing the same issues.

@PragmaFlow I don't have any plans to actively work on yrb-actioncable for the time being. I am willing to guide anyone through the process of fixing things though.

[jeygeethan]

@eliias I think this would be a great addition to the rails community. Can you help me run the project? I would like to where can I being.

[eliias]

@eliias I think this would be a great addition to the rails community. Can you help me run the project? I would like to where can I being.

@jeygeethan We (y-crdt) happily welcome contributors, and I am willing to support, review, and merge PRs. If you have a specific project in mind, just let me know, and we can discuss that (https://calendly.com/iv-call/30min), or feel free to pick up any of the issues from here.

The current state of the gem is more a proof of concept and I personally did never use in a production environment, so there is lots of work to just make it production read. It would be great if you are actually using it in one of our projects.

That said, the most interesting piece of work is getting “Reliable sync” done, which is pretty much a rewrite of everything. There is a WIP PR for this https://github.com/y-crdt/yrb-actioncable/blob/42b0c5f8dc229aef0918ca32d0b801dee8039354/docs/reliable.md, and a related issue for the client #25.