[bitnami/redis] I want to connect to Redis Sentinel from outside k8s.

[sdYoo]

Name and Version

bitnami/redis 17.6.0

What architecture are you using?

amd64

What steps will reproduce the bug?

1. I enabled nodeport in Sentinel settings.

sentinel:
  count: 3
  masterSet: test
  service:
    type: NodePort
    nodePorts:
      redis: "16379"
      sentinel: "26379"

2. Check nodePorts to k8s service

redis-sentinel     NodePort    10.250.70.77    <none>        16379:16379/TCP,26379:26379/TCP

3. When using redis-cli, both redis and sentinel are connected normally.

4. Using java lettuce, we try to connect to the Sentinel k8s domain as follows.
   -> It is possible to connect directly to redis using the host IP.
   -> However, if you connect using Sentinel method, an error occurs.

=== java code
Connect OK : LettuceConnectionFactory(standaloneConfiguration, lettuceClientConfiguration)
Connect Fail : LettuceConnectionFactory(sentinelConfiguration, lettuceClientConfiguration)
=== Connect Fail error
ERROR 78119 --- [nio-8080-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: org.springframework.data.redis.RedisConnectionFailureException: Unable to connect to Redis] with root cause

java.net.UnknownHostException: redis-sentinel-node-0.redis-sentinel-headless.redis-sentinel.svc.cluster.local: nodename nor servname provided, or not known
	at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[na:na]
	at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52) ~[na:na]

5. What settings do I need to add to connect to Sentinel from outside of k8s?

What is the expected behavior?

After activating nodeport, you must be able to connect to the node port or host IP, not the k8s service domain.

java.net.UnknownHostException: test-redis-sentinel-node-1.test-redis-sentinel-headless.redis-sentinel.svc.cluster.local: nodename nor servname provided, or not known

What do you see instead?

I enabled the node port in Sentinel, but it still returns the k8s service address and cannot be accessed from outside k8s.

java.net.UnknownHostException: redis-sentinel-node-0.redis-sentinel-headless.redis-sentinel.svc.cluster.local: nodename nor servname provided, or not known
	at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[na:na]
	at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52) ~[na:na]

Additional information

I would like to know the settings for connecting to redis sentinel from outside without using a separate open source(ex haproxy).

[carrodher]

The issue may not be directly related to the Bitnami container image or Helm chart, but rather to how the application is being utilized or configured in your specific environment.

Having said that, if you think that's not the case and are interested in contributing a solution, we welcome you to create a pull request. The Bitnami team is excited to review your submission and offer feedback. You can find the contributing guidelines here.

Your contribution will greatly benefit the community. Feel free to reach out if you have any questions or need assistance.

If you have any questions about the application itself, customizing its content, or questions about technology and infrastructure usage, we highly recommend that you refer to the forums and user guides provided by the project responsible for the application or technology.

With that said, we'll keep this ticket open until the stale bot automatically closes it, in case someone from the community contributes valuable insights.

[sdYoo]

@carrodher

Thank you for quick reply.
As shown below, the pod address is included in the Sentinel information, so it does not seem to be able to connect from the outside.

===================================
127.0.0.1:36379> sentinel masters
1)  1) "name"
    2) "testv2master"
    3) "ip"
    4) "test-redis-sentinel-node-0.redis-sentinel"
    5) "port"
    6) "30290"
===================================

Even if you enter the node IP in the Sentinel extraEnvVars setting, it is not reflected in the actual Sentinel.
Are there any settings in the chart that can be changed with node IP?

sentinel:
  count: 3
  masterSet: sentinelmaster
  service:
    type: NodePort
    nodePorts:
      redis: "30211"
      sentinel: "30311"
  extraEnvVars:
    - name: REDIS_MASTER_HOST
      value: 10.150.xxx.xxx
    - name: REDIS_MASTER_PORT_NUMBER
      value: "30289"

If I can preserve the IP information returned by Sentinel from "test-redis-sentinel-node-0.redis-sentinel" to "10.150.xxx.xxx",
I think I can connect redis sentinel mode using nodeport outside k8s.

[sdYoo]

@carrodher I went through the settings below
I was able to change the Master IP(10.150.xxx.xxx) in the pod address(test-redis-sentinel-node-0.redis-sentinel).

"externalMaster.enabled: true"

but I am constantly getting logs like the one below in Sentinel...
dns lookup seems to be failing in Sentinel. Is there a way to fix it?
redis works fine.
What I set up is nodeport settings and "externalMaster" activation.

=======
sentinel logs
=======
 Failed to resolve hostname 'test-redis-sentinel-node-2.redis-sentinel'
1:X 25 Mar 2024 09:42:55.601 # Failed to resolve hostname 'test-redis-sentinel-node-1.redis-sentinel'
1:X 25 Mar 2024 09:42:56.164 # Failed to resolve hostname 'test-redis-sentinel-node-1.redis-sentinel'
1:X 25 Mar 2024 09:42:56.165 # Failed to resolve hostname 'test-redis-sentinel-node-1.redis-sentinel'
1:X 25 Mar 2024 09:42:56.165 * +sentinel-invalid-addr sentinel 73c9a7e607afd5af5530c2d5bea794d543f3e0d0 test-redis-sentinel-node-2.redis-sentinel 36379 @ testmaster 10.150.xxx.xxx 30201
1:X 25 Mar 2024 09:42:56.166 # Failed to resolve hostname 'test-redis-sentinel-node-1.redis-sentinel'
1:X 25 Mar 2024 09:42:56.166 * +sentinel sentinel f5769313475c910f16f9fe5fba5ece45922eb9b8 test-redis-sentinel-node-1.redis-sentinel 36379 @ testmaster 10.150.xxx.xxx 30201
1:X 25 Mar 2024 09:42:56.168 * Sentinel new configuration saved on disk

[jkratzvictoria]

I, too, am looking for a solution to connect to Redis Sentinel from outside the K8s cluster. I'm configuring the service type as LoadBalancer and it seems to initially connect but as soon as I try to do any command GET, SET,DEL, etc. it fails. The issue seems to be that sentinel is sending back the internal K8s names to the Redis client, which won't work outside the K8s cluster.

redis: 2024/03/27 22:39:28 osscluster.go:1764: getting command info: redis: cluster has no nodes
redis: 2024/03/27 22:39:28 osscluster.go:1764: getting command info: redis: cluster has no nodes
redis: 2024/03/27 22:39:28 sentinel.go:724: sentinel: discovered new sentinel="redis-badges-node-0.redis-badges-headless.redis.svc.cluster.local:26379" for master="mymaster"
redis: 2024/03/27 22:39:28 sentinel.go:724: sentinel: discovered new sentinel="redis-badges-node-2.redis-badges-headless.redis.svc.cluster.local:26379" for master="mymaster"
redis: 2024/03/27 22:39:29 sentinel.go:724: sentinel: discovered new sentinel="redis-badges-node-1.redis-badges-headless.redis.svc.cluster.local:26379" for master="mymaster"

And then the application is encountering an error trying to do SET operation

dial tcp: lookup redis-badges-node-0.redis-badges-headless.redis.svc.cluster.local: i/o timeout

Is there a way to configure it where it uses IPs instead of host names? We are using Azure CNI model so all ours pods have an IP on our subnet so they are directly accessible.

[fanjlii]

see https://github.com/bitnami/charts/issues/16436#issuecomment-1541911310 add record to /etc/hosts it works fine~

[jkratzvictoria]

That doesn't work for the official Go Redis client; it still tries to do a lookup on the hosts even with the entries in the hosts file.

dial tcp: lookup redis-badges-node-0.redis-badges-headless.redis.svc.cluster.local: i/o timeout

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[OronDF343]

Any solution to this that does not involve editing the hosts file?

[javsalgar]

In other charts we have the externalAccess section, which creates a LoadBalancer service per pod. Would that work for your use case?

[OronDF343]

As others have described, depends on whether that would work with Sentinel as it returns hostnames, not IPs, and I can't use hostnames/IPs that are cluster-internal.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[OronDF343]

Why did the bot close this when there clearly was activity?

[migruiz4]

Hi there,
I have created an internal task to implement the Redis Sentinel external access if possible and added the 'on-hold' label so the stale-bot does not close this issue.

If you would like to contribute by implementing a feature similar to what already exists in the bitnami/redis-cluster chart, feel free to send a PR with your changes and we will be happy to review it and help with anything if needed.

[netscrol2016]

it is actual problem !!!!

[lpellegr]

I am facing the same issue. Whatever you configure, when you get the primary from a sentinel it is either a cluster private IP or a hostname that even if configured with external-dns still resolves as a private pod IP address, even if k8s nodes have a public IP address.

As other have mentioned, would be really great to have a way to get sentinels to return a public IP address. Most probably by deploying a load balancer per primary and using this load balancer IP. That's what I expected when I used primary.service.type: LoadBalancer but that's not the case.

[dannotes]

Any possible solutions for this issues? Will be much appreciated, we are planning to host Redis in AKS and access outside the cluster. We can make the sentinel service in type load balancer and it returns pod private IP which will not be accessible outside the cluster 😔.

[dannotes]

For those seeking a solution, I have addressed the challenge by adding the following parameters to the values.yaml file. This configuration creates an additional container within the Redis pod, updates the label of the master pod with IsMaster=true, and deploys a master service that consistently points to the master node. Although this approach is experimental, as noted in the documentation, I have proceeded with it.

sentinel:
  masterService:
    enabled: true
    type: LoadBalancer
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-internal: "true"
      external-dns.alpha.kubernetes.io/hostname: "redis-master.aks-privatedns.com"
      external-dns.alpha.kubernetes.io/ttl: "60"