Spring Boot / Spring Security App, geting AWS Redis Elasticache production error (failed: Connection reset by peer & CROSSSLOT error)

[dreamstar-enterprises]

Getting following connection error with AWS Elasticache (production)

2024-10-19T08:33:26.374Z INFO 1 --- [BFFApplication] [xecutorLoop-3-4] f.b.a.r.s.RedisSecurityContextRepository : SAVING AUTHORIZATION REQUEST
2024-10-19T08:33:26.379Z INFO 1 --- [BFFApplication] [xecutorLoop-3-4] f.b.a.r.s.RedisSecurityContextRepository : Saved Authorization Request org.springframework.security.oauth2.core.endpoint.OAuth2AuthorizationRequest@117c1dd in WebSession: org.springframework.security.config.web.server.ServerHttpSecurity$OAuth2LoginSpec$OidcSessionRegistryWebFilter$OidcSessionRegistryServerWebExchange$OidcSessionRegistryWebSession@1420e939
2024-10-19T08:33:26.379Z INFO 1 --- [BFFApplication] [xecutorLoop-3-4] f.b.a.r.s.RedisSecurityContextRepository : Authorization Request state: O2Ltdm7yaOIpkBYzAl7k5YY0XktgoAqh3T04kYfIfYI=
2024-10-19T08:33:26.379Z INFO 1 --- [BFFApplication] [xecutorLoop-3-4] c.f.b.a.r.OAuth2ServerRedirectStrategy : RUNNING SERVER REDIRECT STRATEGY: 302 FOUND
2024-10-19T08:33:29.682Z INFO 1 --- [BFFApplication] [or-http-epoll-2] .b.a.r.t.CustomServerCsrfTokenRepository : Loading CSRF token
2024-10-19T08:33:29.684Z INFO 1 --- [BFFApplication] [or-http-epoll-2] c.f.b.a.h.c.SPACsrfTokenRequestHandler : Handling CSRF token: MonoSwitchIfEmpty
2024-10-19T08:33:29.684Z INFO 1 --- [BFFApplication] [or-http-epoll-2] .b.a.r.t.CustomServerCsrfTokenRepository : Generating CSRF token
2024-10-19T08:33:29.685Z INFO 1 --- [BFFApplication] [or-http-epoll-2] f.b.a.r.s.RedisSecurityContextRepository : REMOVING AUTHORIZATION REQUEST
2024-10-19T08:33:29.692Z INFO 1 --- [BFFApplication] [xecutorLoop-3-1] f.b.a.r.s.RedisSecurityContextRepository : Removed authorization request
2024-10-19T08:33:29.692Z INFO 1 --- [BFFApplication] [xecutorLoop-3-1] f.b.a.r.s.RedisSecurityContextRepository : Successfully deserialized Authorization Request: org.springframework.security.oauth2.core.endpoint.OAuth2AuthorizationRequest@705f23d3
io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer
2024-10-19T08:33:29.695Z WARN 1 --- [BFFApplication] [or-http-epoll-1] r.netty.http.client.HttpClientConnect : [04ff363b-2, L:/10.0.141.193:43530 - R:dev-xxxxxxxxxx.auth0.com/104.18.35.70:443] **The connection observed an error, the request cannot be retried as the headers/body were sent**
Caused by: io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer
at java.base/java.lang.Thread.run(Thread.java:831) ~[na:na]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.112.Final.jar!/:4.1.112.Final]

The only message I see is:

The connection observed an error, the request cannot be retried as the headers/body were sent

If it does pass that stage:

I later also get this 2nd error:

2024-10-19T09:13:07.654Z INFO 1 --- [BFFApplication] [or-http-epoll-2] c.f.b.a.m.GrantedAuthoritiesMapperConfig : PROCESSING AUTHORITY: OIDC_USER
2024-10-19T09:13:07.654Z INFO 1 --- [BFFApplication] [or-http-epoll-2] c.f.b.a.m.GrantedAuthoritiesMapperConfig : OidcUserAuthority ID Token Claims
2024-10-19T09:13:07.660Z INFO 1 --- [BFFApplication] [xecutorLoop-3-5] c.f.b.a.s.CustomSessionIdGenerator : Successfully generated session ID: BFF-1729329187658-c73aad09-7fd2-4980-9fc5-0c2704e97ec4
org.springframework.data.redis.RedisSystemException: Error in execution
Caused by: io.lettuce.core.RedisCommandExecutionException: CROSSSLOT Keys in request don't hash to the same slot

The only message I see here is:

CROSSSLOT Keys in request don't hash to the same slot

Local Laptop Development runs fine (using Redis.com cache)

2024-10-19T09:49:51.937+01:00  INFO 96570 --- [BFFApplication] [xecutorLoop-3-2] f.b.a.r.s.RedisSecurityContextRepository : SAVING AUTHORIZATION REQUEST
2024-10-19T09:49:51.938+01:00  INFO 96570 --- [BFFApplication] [xecutorLoop-3-2] c.f.b.a.r.OAuth2ServerRedirectStrategy   : RUNNING SERVER REDIRECT STRATEGY: 302 FOUND
2024-10-19T09:49:51.938+01:00  INFO 96570 --- [BFFApplication] [xecutorLoop-3-2] f.b.a.r.s.RedisSecurityContextRepository : Authorization Request state: 81PGq8kSVgHdOxfqxj_awoELdV5qvFwu50xKuyHSICI=
2024-10-19T09:49:51.938+01:00  INFO 96570 --- [BFFApplication] [xecutorLoop-3-2] f.b.a.r.s.RedisSecurityContextRepository : Saved Authorization Request org.springframework.security.oauth2.core.endpoint.OAuth2AuthorizationRequest@14e4bf33 in WebSession: org.springframework.security.config.web.server.ServerHttpSecurity$OAuth2LoginSpec$OidcSessionRegistryWebFilter$OidcSessionRegistryServerWebExchange$OidcSessionRegistryWebSession@6fa54c91
2024-10-19T09:49:55.569+01:00  INFO 96570 --- [BFFApplication] [ctor-http-nio-2] .b.a.r.t.CustomServerCsrfTokenRepository : Loading CSRF token
2024-10-19T09:49:55.569+01:00  INFO 96570 --- [BFFApplication] [ctor-http-nio-2] .b.a.r.t.CustomServerCsrfTokenRepository : Generating CSRF token
2024-10-19T09:49:55.570+01:00  INFO 96570 --- [BFFApplication] [ctor-http-nio-2] c.f.b.a.h.c.SPACsrfTokenRequestHandler   : Handling CSRF token: MonoSwitchIfEmpty
2024-10-19T09:49:55.574+01:00  INFO 96570 --- [BFFApplication] [ctor-http-nio-2] f.b.a.r.s.RedisSecurityContextRepository : REMOVING AUTHORIZATION REQUEST
2024-10-19T09:49:55.610+01:00  INFO 96570 --- [BFFApplication] [xecutorLoop-3-8] f.b.a.r.s.RedisSecurityContextRepository : Successfully deserialized Authorization Request: org.springframework.security.oauth2.core.endpoint.OAuth2AuthorizationRequest@40a82415
2024-10-19T09:49:55.610+01:00  INFO 96570 --- [BFFApplication] [xecutorLoop-3-8] f.b.a.r.s.RedisSecurityContextRepository : Removed authorization request
2024-10-19T09:49:56.746+01:00  INFO 96570 --- [BFFApplication] [ctor-http-nio-4] c.f.b.a.m.GrantedAuthoritiesMapperConfig : PROCESSING AUTHORITY: OIDC_USER
2024-10-19T09:49:56.747+01:00  INFO 96570 --- [BFFApplication] [ctor-http-nio-4] c.f.b.a.m.GrantedAuthoritiesMapperConfig : OidcUserAuthority ID Token Claims

There is an old issue on Github that talks about the 1st error:

reactor/reactor-netty#1774

The 2nd error is due to some Redis Cluster config issue, but I can't find any Spring documentation about it.

Here is my Redis Connection factory config, but I can't see where I am going wrong

@Configuration
internal class RedisConnectionFactoryConfig(
    private val springDataProperties: SpringDataProperties,
    private val profileProperties: ProfileProperties
) {

    // reactive RedisConnectionFactory for key expiration event handling
    @Bean
    @Primary
    fun reactiveRedisConnectionFactory(): ReactiveRedisConnectionFactory {

        // configure Redis standalone configuration
        val config = RedisStandaloneConfiguration()
        config.hostName = springDataProperties.redis.host
        config.port = springDataProperties.redis.port
        config.setPassword(RedisPassword.of(springDataProperties.redis.password))

        // create client options

        // Create SSL options if SSL is required
        val sslOptions = SslOptions.builder()
            .jdkSslProvider()  // Or use OpenSslProvider if you prefer
            .build()

        // Create timeout options
        val timeoutOptions = TimeoutOptions.builder()
            .fixedTimeout(Duration.ofSeconds(20))
            .timeoutCommands(true)
            .build()

        // cluster specific settings for optimal reliability.
        val clusterTopologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
            .enablePeriodicRefresh(Duration.ofSeconds(5))
            .dynamicRefreshSources(false)
            .adaptiveRefreshTriggersTimeout(Duration.ofSeconds(5))
            .enableAllAdaptiveRefreshTriggers().build()

        // create socket options
        val socketOptions = SocketOptions.builder()
            .keepAlive(SocketOptions.DEFAULT_SO_KEEPALIVE)
            .tcpNoDelay(SocketOptions.DEFAULT_SO_NO_DELAY)
            // time to wait for connection to be established, before considering it as a failed connection
            .connectTimeout(Duration.ofSeconds(60))
            .build()

        val mappingFunction: (HostAndPort) -> HostAndPort = { hostAndPort ->
            val host = springDataProperties.redis.host
            val addresses: Array<InetAddress> = try {
                DnsResolvers.JVM_DEFAULT.resolve(host)
            } catch (e: UnknownHostException) {
                e.printStackTrace()
                emptyArray() // Handle error and return an empty array
            }

            val cacheIP = addresses.firstOrNull()?.hostAddress ?: ""
            var finalAddress = hostAndPort

            if (hostAndPort.hostText == cacheIP) {
                finalAddress = HostAndPort.of(host, hostAndPort.port)
            }

            finalAddress
        }

        val resolver = MappingSocketAddressResolver.create(DnsResolvers.JVM_DEFAULT, mappingFunction)

        // customize thread pool size
        val clientResources = DefaultClientResources.builder()
            .ioThreadPoolSize(8)
            .computationThreadPoolSize(8)
            .socketAddressResolver(resolver)
            .build()

        val clusterClientOptionsBuilder = ClusterClientOptions.builder()
            .autoReconnect(true)
            .pingBeforeActivateConnection(true)
            .timeoutOptions(timeoutOptions)
            .socketOptions(socketOptions)
            .topologyRefreshOptions(clusterTopologyRefreshOptions)
            .validateClusterNodeMembership(true)
            .suspendReconnectOnProtocolFailure(true)
            .disconnectedBehavior(DEFAULT_DISCONNECTED_BEHAVIOR)
            .decodeBufferPolicy(DecodeBufferPolicies.ratio(0.5F))
            .requestQueueSize(1000)
            .maxRedirects(DEFAULT_MAX_REDIRECTS)
            .publishOnScheduler(true) //DEFAULT_PUBLISH_ON_SCHEDULER.
            .protocolVersion(ProtocolVersion.RESP3) // Use RESP3 Protocol to ensure AUTH command is used for handshake.

        // conditionally use sslOptions if profileProperties.active is 'prod'
        if (profileProperties.active == "prod") {
            clusterClientOptionsBuilder.sslOptions(sslOptions)
        }

        // build the clientClusterOptions configuration
        val clusterClientOptions = clusterClientOptionsBuilder.build()

        // configure connection pool settings
        fun buildLettucePoolConfig(): GenericObjectPoolConfig<Any> {
            val poolConfig = GenericObjectPoolConfig<Any>()
            poolConfig.maxTotal = 100
            poolConfig.maxIdle = 50
            poolConfig.minIdle = 10
            poolConfig.setMaxWait(Duration.ofSeconds(120))
            poolConfig.timeBetweenEvictionRuns = Duration.ofSeconds(120)
            poolConfig.minEvictableIdleTime = Duration.ofMinutes(5)
            poolConfig.testOnBorrow = true
            poolConfig.testWhileIdle = true
            poolConfig.testOnReturn = true
            poolConfig.blockWhenExhausted = true
            poolConfig.lifo = true
            return poolConfig
        }

        // create Lettuce client configuration with authentication details
        val clientConfigBuilder = LettucePoolingClientConfiguration.builder()
            // maximum time allowed for a Redis command to execute before the operation is considered timed out.
            .commandTimeout(Duration.ofSeconds(60))
            .clientResources(clientResources)
            .clientOptions(clusterClientOptions)
            .poolConfig(buildLettucePoolConfig())

        // conditionally enable SSL only if profileProperties.active is 'prod'
        if (profileProperties.active == "prod") {
            clientConfigBuilder.useSsl()
        }

        // build the clientConfig configuration
        val clientConfig = clientConfigBuilder.build()

        // create Lettuce connection factory
        return LettuceConnectionFactory(config, clientConfig).apply {
            afterPropertiesSet()
            validateConnection = false
            setShareNativeConnection(true)
        }
    }

}

Can someone help?