Reaper cannot connect to IPv6 addresses

[andreasgeisslerdt]

Project board link

We deployed a cassandra cluster in a pure IPv6 environment using k8ssandra-operator.
But I needed to set the JAVA options:

        additionalOptions:
          - "-Djava.net.preferIPv4Stack=false"
          - "-Djava.net.preferIPv6Addresses=true"
          - "-Djava.net.preferIPv6Stack=true"
        additionalJvmServerOptions:
          - "-Djava.net.preferIPv6Addresses=true"
          - "-Djava.net.preferIPv6Stack=true"
          - "-Djava.net.preferIPv4Stack=false"

But Reaper startup fails with connection issues:

INFO   [2024-12-10 14:08:17,254] [main] c.d.d.core - DataStax Java driver 3.11.0 for Apache Cassandra 
INFO   [2024-12-10 14:08:17,257] [main] c.d.d.c.GuavaCompatibility - Detected Guava >= 19 in the classpath, using modern compatibility layer 
INFO   [2024-12-10 14:08:17,279] [main] c.d.d.c.Native - Could not load JNR C Library, native system calls through this library will not be available (set this logger level to DEBUG to see the full stack trace). 
INFO   [2024-12-10 14:08:17,279] [main] c.d.d.c.ClockFactory - Using java.lang.System clock to generate timestamps. 
INFO   [2024-12-10 14:08:17,421] [main] c.d.d.c.NettyUtil - Did not find Netty's native epoll transport in the classpath, defaulting to NIO. 
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cassandra-dc1-service/2a01:598:7e0:16f:0:0:0:58:9042 (com.datastax.driver.core.exceptions.TransportException: [cassandra-dc1-service/2a01:598:7e0:16f:0:0:0:58:9042] Connection has been closed), cassandra-dc1-service/2a01:598:7e0:16f:0:0:0:3f7:9042 (com.datastax.driver.core.exceptions.TransportException: [cassandra-dc1-service/2a01:598:7e0:16f:0:0:0:3f7:9042] Connection has been closed), cassandra-dc1-service/2a01:598:7e0:16f:0:0:0:327:9042 (com.datastax.driver.core.exceptions.TransportException: [cassandra-dc1-service/2a01:598:7e0:16f:0:0:0:327:9042] Connection has been closed), cassandra-dc1-service/2a01:598:7e0:16f:0:0:0:10d:9042, cassandra-dc1-service/2a01:598:7e0:16f:0:0:0:16:9042 [only showing errors of first 3 hosts, use getErrors() for more details])
	at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:270)
	at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:109)
	at com.datastax.driver.core.Cluster$Manager.negotiateProtocolVersionAndConnect(Cluster.java:1813)
	at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1726)
	at com.datastax.driver.core.Cluster.init(Cluster.java:214)
	at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:387)
	at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:366)
	at com.datastax.driver.core.Cluster.connect(Cluster.java:311)
	at systems.composable.dropwizard.cassandra.CassandraHealthCheck.<init>(CassandraHealthCheck.java:42)
	at systems.composable.dropwizard.cassandra.CassandraFactory.build(CassandraFactory.java:504)
	at systems.composable.dropwizard.cassandra.CassandraFactory.build(CassandraFactory.java:447)
	at io.cassandrareaper.storage.cassandra.CassandraStorageFacade.<init>(CassandraStorageFacade.java:124)
	at io.cassandrareaper.storage.InitializeStorage.initializeStorageBackend(InitializeStorage.java:68)
	at io.cassandrareaper.ReaperDbMigrationCommand.run(ReaperDbMigrationCommand.java:53)
	at io.cassandrareaper.ReaperDbMigrationCommand.run(ReaperDbMigrationCommand.java:30)
	at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:98)
	at io.dropwizard.cli.Cli.run(Cli.java:78)
	at io.dropwizard.Application.run(Application.java:94)
	at io.cassandrareaper.ReaperApplication.main(ReaperApplication.java:100)

This might be related to the the known issue in the datastax driver (here 3.11.0):
https://stackoverflow.com/questions/77714194/cassandra-datastax-ipv6-connection

Additionally in k8ssandra-operator CRD I am not able to set the Java Options.

┆Issue is synchronized with this Jira Story by Unito
┆Issue Number: REAP-196

[andreasgeisslerdt]

I found the reason for the problem and close the issue.
Problem was a wrong configured Istio setting of the cassandra pods, which enabled the internal communication between the cassandra pods, but blocked the external communication.
Sorry for the inconveniences and best regards
Andreas