Sporadic exception when using this gem

[krschacht]

I changed my adapter from redis to this gem's enhanced_postgresql and tried the app. At first it worked! I was excited but as I'm testing it more I get random failures with this exception:

base-1      | #<Thread:0x0000ffffa5e6a9b0 /usr/local/bundle/gems/actioncable-7.1.3.2/lib/action_cable/subscription_adapter/postgresql.rb:81 run> terminated with exception (report_on_exception is true):
base-1      | /usr/local/bundle/gems/activesupport-7.1.3.2/lib/active_support/messages/codec.rb:57:in `catch_and_raise': ActiveSupport::MessageEncryptor::InvalidMessage
base-1      |   from /usr/local/bundle/gems/activesupport-7.1.3.2/lib/active_support/message_encryptor.rb:242:in `decrypt_and_verify'
base-1      |   from /usr/local/bundle/gems/actioncable-enhanced-postgresql-adapter-1.0.1/lib/action_cable/subscription_adapter/enhanced_postgresql.rb:118:in `invoke_callback'
base-1      |   from /usr/local/bundle/gems/actioncable-7.1.3.2/lib/action_cable/subscription_adapter/subscriber_map.rb:43:in `block in broadcast'
base-1      |   from /usr/local/bundle/gems/actioncable-7.1.3.2/lib/action_cable/subscription_adapter/subscriber_map.rb:42:in `each'
base-1      |   from /usr/local/bundle/gems/actioncable-7.1.3.2/lib/action_cable/subscription_adapter/subscriber_map.rb:42:in `broadcast'
base-1      |   from /usr/local/bundle/gems/actioncable-7.1.3.2/lib/action_cable/subscription_adapter/postgresql.rb:106:in `block (4 levels) in listen'
base-1      |   from /usr/local/bundle/gems/actioncable-7.1.3.2/lib/action_cable/subscription_adapter/postgresql.rb:105:in `wait_for_notify'
base-1      |   from /usr/local/bundle/gems/actioncable-7.1.3.2/lib/action_cable/subscription_adapter/postgresql.rb:105:in `block (3 levels) in listen'
base-1      |   from /usr/local/bundle/gems/actioncable-7.1.3.2/lib/action_cable/subscription_adapter/postgresql.rb:90:in `loop'
base-1      |   from /usr/local/bundle/gems/actioncable-7.1.3.2/lib/action_cable/subscription_adapter/postgresql.rb:90:in `block (2 levels) in listen'
base-1      |   from /usr/local/bundle/gems/actioncable-7.1.3.2/lib/action_cable/subscription_adapter/postgresql.rb:89:in `catch'
base-1      |   from /usr/local/bundle/gems/actioncable-7.1.3.2/lib/action_cable/subscription_adapter/postgresql.rb:89:in `block in listen'
base-1      |   from /usr/local/bundle/gems/actioncable-7.1.3.2/lib/action_cable/subscription_adapter/postgresql.rb:45:in `with_subscriptions_connection'
base-1      |   from /usr/local/bundle/gems/actioncable-enhanced-postgresql-adapter-1.0.1/lib/action_cable/subscription_adapter/enhanced_postgresql.rb:83:in `with_subscriptions_connection'
base-1      |   from /usr/local/bundle/gems/actioncable-7.1.3.2/lib/action_cable/subscription_adapter/postgresql.rb:88:in `listen'
base-1      |   from /usr/local/bundle/gems/actioncable-7.1.3.2/lib/action_cable/subscription_adapter/postgresql.rb:83:in `block in initialize'
base-1      | Exiting

I'm trying to make sense of the source within enhanced_postgresql.rb but I don't have any ideas yet as to why this fails occasionally.

[krschacht]

I've spent more time trying to debug this. I normally run my development environment within a simple Docker setup in my Mac. When I try running my app with this gem outside of Docker, then I don't get this exception. I have not been able to isolate any difference between the two which would explain it. I'll paste my Docker setup below in case you'd like to examine it.

In addition, when I am running my app within Docker, I get this exception the first time it attempts to send a turbo stream response. All subsequent times there is no exception, but there also is no successful streaming. If I kill the app and restart it, then I again get this exception on the first attempt and all subsequent attempts fail silently.

My Dockerfile

ARG RUBY_VERSION=3.2.3
FROM ruby:${RUBY_VERSION}-alpine AS development

RUN apk add --no-cache git build-base postgresql-dev curl-dev gcompat tzdata vips-dev imagemagick

ENV BUNDLE_CACHE=/tmp/bundle \
  BUNDLE_JOBS=2 \
  PORT=3000

WORKDIR /rails
COPY Gemfile Gemfile.lock .ruby-version ./

RUN --mount=type=cache,id=gems,target=/tmp/bundle \
  bundle install

RUN apk add --no-cache postgresql-client

ENTRYPOINT ["/rails/bin/docker-entrypoint"]
CMD ["./bin/rails", "server", "-b", "0.0.0.0"]

And my docker-compose

version: '3.1'

services:
  postgres:
    image: postgres:16
    restart: always
    environment:
      POSTGRES_USER: app
      POSTGRES_DB: app_development
      POSTGRES_PASSWORD: secret
    volumes:
    - hostedgpt_pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "app", "-d", "app_development"]
      interval: 1s
      retries: 5

  base: &base
    depends_on: ["postgres"]
    restart: on-failure:3
    tty: true
    build:
      context: .
      target: development
    environment:
      - DATABASE_URL=postgres://app:secret@postgres/app_development
      - DEV_HOST=${DEV_HOST:-localhost} # Set if you want to use a different hostname
    ports: ["3000:3000"]
    volumes:
      - .:/rails
      - /rails/tmp

  worker:
    <<: *base
    restart: on-failure:3
    tty: false
    ports: []
    command: bin/rails solid_queue:start

volumes:
  hostedgpt_pgdata:

[krschacht]

I have worked around my issue. It has something to do with how secret_key_base is being determined. I was using secret_key_base from my credentials.enc file. When I deleted that file and switched to using it from SECRET_KEY_BASE then things worked fine.

My hunch is that enhanced_postgresql.rb is reading Rails.application.secret_key_base rather than Rails.application.credentials.config[:secret_key_base]? When I added some debugging I noticed those were returning different values and I believe it's the one from credentials that I need to be using. But again, this was only an issue within my Docker and not when I do bin/dev.

I don't know enough but I'll leave this issue here for you in case it's helpful. I won't investigate any further since I've worked around the issue. Feel free to close the issue since it's probably not important for you to get to the bottom of it.

And BTW, I greatly appreciate you creating this gem and sharing it. It's a big help for the project I'm working on and I don't know how long it will take for rails itself to work around this issue. Thank you!

[lachappers]

I have worked around my issue. It has something to do with how secret_key_base is being determined. I was using secret_key_base from my credentials.enc file. When I deleted that file and switched to using it from SECRET_KEY_BASE then things worked fine.

My hunch is that enhanced_postgresql.rb is reading Rails.application.secret_key_base rather than Rails.application.credentials.config[:secret_key_base]? When I added some debugging I noticed those were returning different values and I believe it's the one from credentials that I need to be using. But again, this was only an issue within my Docker and not when I do bin/dev.

I don't know enough but I'll leave this issue here for you in case it's helpful. I won't investigate any further since I've worked around the issue. Feel free to close the issue since it's probably not important for you to get to the bottom of it.

And BTW, I greatly appreciate you creating this gem and sharing it. It's a big help for the project I'm working on and I don't know how long it will take for rails itself to work around this issue. Thank you!

Hello!
How did you work around this? Is it something to do with setting payload_encryptor_secret somewhere?

[krschacht]

It had something to do with a discrepancy between these two values: Rails.application.secret_key_base Rails.application.credentials.config[:secret_key_base] I added debugging at the line that was throwing the exception and these two values were different, and I don’t think they are ever supposed to be. And it looks like your code was reading from application rather than credentials. I wiped my whole app setup, deleted my credentials file, and set SECRET KEY BASE in the environment instead and then those values matched. This had never impacted any other aspect of my setup and I don’t fully understand rails encryption. But that’s the best clue I have. Sorry I don’t have a full fix!

…

On Tue, May 7, 2024 at 12:22 PM lachappers ***@***.***> wrote: I have worked around my issue. It has something to do with how secret_key_base is being determined. I was using secret_key_base from my credentials.enc file. When I deleted that file and switched to using it from SECRET_KEY_BASE then things worked fine. My hunch is that enhanced_postgresql.rb is reading Rails.application.secret_key_base rather than Rails.application.credentials.config[:secret_key_base]? When I added some debugging I noticed those were returning different values and I believe it's the one from credentials that I need to be using. But again, this was only an issue within my Docker and not when I do bin/dev. I don't know enough but I'll leave this issue here for you in case it's helpful. I won't investigate any further since I've worked around the issue. Feel free to close the issue since it's probably not important for you to get to the bottom of it. And BTW, I greatly appreciate you creating this gem and sharing it. It's a big help for the project I'm working on and I don't know how long it will take for rails itself to work around this issue. Thank you! Hello! How did you work around this? Is it something to do with setting payload_encryptor_secret in solid_queue.yml? — Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAIR5KDPI6FUUTOOZELVIDZBEEUTAVCNFSM6AAAAABHHZB5QSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJYHE2DKNJZGM> . You are receiving this because you authored the thread.Message ID: <reclaim-the-stack/actioncable-enhanced-postgresql-adapter/issues/2/2098945593 @github.com>