feat(scylladb): add scylladb provider #2919

DanielHe4rt · 2024-12-13T16:29:45Z

What does this PR do?

Implements the Database ScyllaDB as a new provider.

Under the ScyllaDBContainer you will have the possibility of:

Enable Shard Awareness port using WithShardAwareness;
Set Docker Start Commands with WithCommand
Fetch ports with ConnectionHost
Enable ScyllaDB Alternator Endpoint (DynamoDB Compatible API)

API Example

ctx := context.Background()

ctr, err := scylladb.Run(ctx,
   "scylladb/scylla:6.2",
   scylladb.WithConfigFile(filepath.Join("testdata", "scylla.yaml")),
   scylladb.WithShardAwareness(),
   //...
)

Why is it important?

ScyllaDB is a NoSQL Database that is being constantly used by big companies and acts as a Drop-in Replacement to CassandraDB and DynamoDB.

How to test this PR

Clone the repository and enter the scylladb module folder:

git clone https://github.com/basementdevs/testcontainers-go basement-testcontainers-go
cd basement-testcontainers-go/modules/scylladb
make test

netlify · 2024-12-13T16:30:06Z

✅ Deploy Preview for testcontainers-go ready!

Name	Link
🔨 Latest commit	`82e5352`
🔍 Latest deploy log	https://app.netlify.com/sites/testcontainers-go/deploys/6762f709f7fcbb0008adf34d
😎 Deploy Preview	https://deploy-preview-2919--testcontainers-go.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

…wareness

…rd-awareness port

stevenh

Thanks for a PR @DanielHe4rt, have done an initial review with some suggestions and questions for you.

stevenh · 2024-12-15T12:34:20Z

modules/scylladb/scylladb.go

+	shardAwarePort = "19042/tcp"
+)
+
+type Container struct {


suggestion: Add a doc comment to describe.

stevenh · 2024-12-15T12:38:06Z

modules/scylladb/scylladb.go

+		cf := testcontainers.ContainerFile{
+			HostFilePath:      configFile,
+			ContainerFilePath: "/etc/scylla/scylla.yaml",
+			FileMode:          0o755,


suggestion: no need for the file to be executable

modules/scylladb/scylladb.go

stevenh · 2024-12-15T12:43:42Z

modules/scylladb/scylladb.go

+// WithAlternator enables the Alternator (DynamoDB Compatible API) service in the ScyllaDB container.
+// It will set the "alternator-port" parameter to the specified port.
+// It will also set the "alternator-write-isolation" parameter to "always" as a command line argument to the container.
+func WithAlternator(alternatorPort int) testcontainers.CustomizeRequestOption {


suggestion: use the in type which matches a port definition

Suggested change

func WithAlternator(alternatorPort int) testcontainers.CustomizeRequestOption {

func WithAlternator(alternatorPort uint16) testcontainers.CustomizeRequestOption {

I think having WithAlternator() would be enough, simplifying user setup. Regarding to alternator-write-isolation, is always recommended for testing?

depend on the application. for testing it's "slower" but you probably wont be doing stress testing in the cluster.

stevenh · 2024-12-15T12:44:57Z

modules/scylladb/scylladb.go

+// ConnectionHost returns the host and port of the Scylladb container with the default port
+// and obtaining the host and exposed port from the container
+// Eg: "host:port" -> "localhost:9042" -> "localhost:19042" -> "localhost:8000"
+func (c Container) ConnectionHost(ctx context.Context, port int) (string, error) {


suggestion: use uint16 to match port requirement.

Suggested change

func (c Container) ConnectionHost(ctx context.Context, port int) (string, error) {

func (c Container) ConnectionHost(ctx context.Context, port uint16) (string, error) {

stevenh · 2024-12-15T12:50:50Z

modules/scylladb/scylladb_test.go

+	testcontainers.CleanupContainer(t, ctr)
+	require.NoError(t, err)
+
+	t.Run("test without shard awareness", func(t *testing.T) {


suggestion: we're standardising on simple names without spaces to ensure they match the output

Suggested change

t.Run("test without shard awareness", func(t *testing.T) {

t.Run("without-shard-awareness", func(t *testing.T) {

More below

stevenh · 2024-12-15T12:53:39Z

modules/scylladb/scylladb_test.go

+
+	ctr, err := scylladb.Run(ctx,
+		"scylladb/scylla:6.2",
+		scylladb.WithConfigFile(filepath.Join("testdata", "scylla.yaml")),


suggestion: with the change to reader, you can leverage embed

//go:embed testdata/scylla.yaml var testConfig []byte

Suggested change

scylladb.WithConfigFile(filepath.Join("testdata", "scylla.yaml")),

scylladb.WithConfig(bytes.NewReader(testConfig)),

stevenh · 2024-12-15T12:54:35Z

modules/scylladb/scylladb_test.go

+	})
+}
+
+func TestScyllaWithAlternator(t *testing.T) {


question: These next two tests look like duplicates of the above, or could be combined?

stevenh · 2024-12-15T12:55:50Z

modules/scylladb/scylladb_test.go

+	return dynamodb.NewFromConfig(cfg, dynamodb.WithEndpointResolverV2(&scyllaAlternatorResolver{HostPort: hostPort})), errors.Join(errs...)
+}
+
+func createTable(client *dynamodb.Client) error {


suggestion: pass in t and require.NoError as thats the only use of this.

stevenh · 2024-12-15T12:56:16Z

modules/scylladb/testdata/scylla.yaml

+murmur3_partitioner_ignore_msb_bits: 12
+strict_is_not_null_in_views: true
+maintenance_socket: ignore
+enable_tablets: true


suggestion: add trailing newline.

eddumelendez · 2024-12-16T22:06:46Z

modules/scylladb/scylladb_test.go

@@ -0,0 +1,194 @@
+package scylladb_test


Can we include a test using ssl? I think would be nice to have.

eddumelendez · 2024-12-17T22:04:35Z

modules/scylladb/scylladb_test.go

+		host, err := ctr.ConnectionHost(ctx, 19042)
+		require.NoError(t, err)
+
+		cluster := gocql.NewCluster(host)
+		session, err := cluster.CreateSession()
+		require.NoError(t, err)
+		defer session.Close()
+
+		var address string
+		err = session.Query("SELECT address FROM system.clients").Scan(&address)
+		require.NoError(t, err)
+	})
+
+	t.Run("test with shard awareness", func(t *testing.T) {
+		host, err := ctr.ConnectionHost(ctx, 19042)
+		require.NoError(t, err)
+
+		cluster := gocql.NewCluster(host)
+		session, err := cluster.CreateSession()
+		require.NoError(t, err)
+		defer session.Close()
+
+		var address string
+		err = session.Query("SELECT address FROM system.clients").Scan(&address)
+		require.NoError(t, err)
+	})


I see both tests are the same. should one of them use 9042 instead? Is it enough testing the connection?

eddumelendez · 2024-12-17T22:10:21Z

modules/scylladb/scylladb.go

+}
+
+// WithShardAwareness enable shard-awareness in the ScyllaDB container so you can use the `19042` port.
+func WithShardAwareness() testcontainers.CustomizeRequestOption {


scylladb starts both ports 9042 and 19042 by default. IMO we should expose a fn to get the mapped port for 19042 and omit this one.

Co-authored-by: Steven Hartland <[email protected]>

DanielHe4rt requested a review from a team as a code owner December 13, 2024 16:29

DanielHe4rt and others added 17 commits December 13, 2024 13:33

feat(scylladb): add base packages

672e6a7

feat(scylladb): container implementation with file config and shard-a…

41ed131

…wareness

feat(scylladb): container base test implementation with files and sha…

0003747

…rd-awareness port

feat(scylladb): add makefile test runner command

6a472cb

feat(scylladb): add alternator integration

264d462

feat(scylladb): add WithCustomCommand and code cleanup

680c279

feat(scylladb): implementation cleanup with test coverage

a995a1c

docs(scylladb): add scylladb base documentation module

8f33187

docs: add links to the version it was introduced

7559c61

chore: add scaffolding files

75de222

chore: rename to Container

37ca8fc

chore: run mod tidy

7f7a476

fix: wrong state after code generation

e780e4f

chore: concat strings

95ca13f

chore: use int

01a4fef

chore: calculate variable once

4388885

chore: simplify types

c0f97ce

DanielHe4rt force-pushed the main branch from 9d50ec1 to c0f97ce Compare December 13, 2024 16:33

stevenh requested changes Dec 15, 2024

View reviewed changes

eddumelendez reviewed Dec 16, 2024

View reviewed changes

eddumelendez reviewed Dec 17, 2024

View reviewed changes

eddumelendez mentioned this pull request Dec 17, 2024

Add scylladb testcontainers/testcontainers-java#8002

Open

chore(scylladb): WithConfig standardising

82e5352

Co-authored-by: Steven Hartland <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(scylladb): add scylladb provider #2919

feat(scylladb): add scylladb provider #2919

DanielHe4rt commented Dec 13, 2024

netlify bot commented Dec 13, 2024 •

edited

Loading

stevenh left a comment

stevenh Dec 15, 2024

stevenh Dec 15, 2024

stevenh Dec 15, 2024

eddumelendez Dec 16, 2024

DanielHe4rt Dec 18, 2024

stevenh Dec 15, 2024

stevenh Dec 15, 2024

stevenh Dec 15, 2024

stevenh Dec 15, 2024

stevenh Dec 15, 2024

stevenh Dec 15, 2024

eddumelendez Dec 16, 2024

eddumelendez Dec 17, 2024 •

edited

Loading

eddumelendez Dec 17, 2024

	func WithAlternator(alternatorPort int) testcontainers.CustomizeRequestOption {
	func WithAlternator(alternatorPort uint16) testcontainers.CustomizeRequestOption {

	func (c Container) ConnectionHost(ctx context.Context, port int) (string, error) {
	func (c Container) ConnectionHost(ctx context.Context, port uint16) (string, error) {

	t.Run("test without shard awareness", func(t *testing.T) {
	t.Run("without-shard-awareness", func(t *testing.T) {

	scylladb.WithConfigFile(filepath.Join("testdata", "scylla.yaml")),
	scylladb.WithConfig(bytes.NewReader(testConfig)),

feat(scylladb): add scylladb provider #2919

Are you sure you want to change the base?

feat(scylladb): add scylladb provider #2919

Conversation

DanielHe4rt commented Dec 13, 2024

What does this PR do?

API Example

Why is it important?

How to test this PR

netlify bot commented Dec 13, 2024 • edited Loading

✅ Deploy Preview for testcontainers-go ready!

stevenh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eddumelendez Dec 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

netlify bot commented Dec 13, 2024 •

edited

Loading

eddumelendez Dec 17, 2024 •

edited

Loading