Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support reading Docker host from the current Docker context #2810

Open
wants to merge 56 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
6c1b3d8
feat: support reading Docker context
mdelapenya Oct 2, 2024
415c896
chore: move read Docker config to the core
mdelapenya Oct 2, 2024
1e620f1
chore: read docker config more consistently
mdelapenya Oct 2, 2024
ea44c92
chore: extract to a parse function to detect remote docker hosts
mdelapenya Oct 2, 2024
549b1cb
docs: add Docker context to docs
mdelapenya Oct 2, 2024
fbefbdf
fix: lint
mdelapenya Oct 2, 2024
1ca3f63
chore: simplify container provider initialisation
mdelapenya Oct 3, 2024
7e1e4b5
feat: support configuring the bridge network name
mdelapenya Oct 3, 2024
2decbe3
chore: rename function
mdelapenya Oct 3, 2024
c596e37
fix: remove wrong copy&paste
mdelapenya Oct 3, 2024
ece6ea5
chore: simplify
mdelapenya Oct 3, 2024
a775c4b
chore: simplify error
mdelapenya Oct 3, 2024
5ac2263
chore: simple naming in test
mdelapenya Oct 3, 2024
932fce2
fix: do not lose the original error
mdelapenya Oct 3, 2024
9c048f7
fix: line endings
mdelapenya Oct 3, 2024
77cdda6
fix: case insensitive regex
mdelapenya Oct 4, 2024
7eab89f
chore: deprecate reaper_network and always use the bridge one
mdelapenya Oct 4, 2024
1e58328
chore: support replacing the bridge network in the endpoint modifier
mdelapenya Oct 4, 2024
8abfeba
fix: lint
mdelapenya Oct 4, 2024
203d762
chore: assume the container runtime uses the default network by default
mdelapenya Oct 4, 2024
464bc9b
chore: rename variable to avoid shading package
mdelapenya Oct 5, 2024
37a8142
fix: if the properties file does not exist, use default config
mdelapenya Oct 5, 2024
07003ff
fix: typo
mdelapenya Oct 5, 2024
adf0ba1
fix: use default config in tests
mdelapenya Oct 7, 2024
b31da78
docs: simplify
mdelapenya Oct 14, 2024
a38a345
chore: simplify error when docker context is not found
mdelapenya Oct 14, 2024
14aa3dd
fix: check if the docker socket is listening
mdelapenya Oct 16, 2024
b327978
docs: document the docker context support
mdelapenya Oct 16, 2024
33528c6
chore: remove bridge network custom configuration
mdelapenya Oct 16, 2024
ca07e04
Merge branch 'main' into context-support
mdelapenya Oct 18, 2024
dab5614
Merge branch 'main' into context-support
mdelapenya Oct 21, 2024
a313dc5
chore: use t.Helper
mdelapenya Oct 21, 2024
5511d17
Merge branch 'main' into context-support
mdelapenya Oct 28, 2024
b38e857
Merge branch 'main' into context-support
mdelapenya Nov 19, 2024
bc2f04b
chore: proper deprecation path
mdelapenya Nov 19, 2024
dc571d3
Merge branch 'main' into context-support
mdelapenya Nov 25, 2024
ad79b43
chore: comment out empty field used for documentation
mdelapenya Nov 25, 2024
4d4769f
chore: use require
mdelapenya Nov 25, 2024
56d6774
docs: update podman commands
mdelapenya Nov 25, 2024
1273e02
chore: readability in var comments
mdelapenya Nov 25, 2024
e7dd83c
chore: unique parsing error messages
mdelapenya Nov 25, 2024
584f6da
docs: readability as function comments
mdelapenya Nov 25, 2024
3b59086
chore: idiomatic var declaration
mdelapenya Nov 25, 2024
1c93fe8
chore: extract file name to constant
mdelapenya Nov 25, 2024
92d306f
chore: simplify default config
mdelapenya Nov 25, 2024
499a05d
chore: make errors private
mdelapenya Nov 25, 2024
cdb874d
chore: simplify function
mdelapenya Nov 25, 2024
1933f42
fix: remove unused func
mdelapenya Nov 25, 2024
f4783a7
chore: bubble up config errors when when config is incorrect
mdelapenya Dec 4, 2024
28f0802
chore: use better test names
mdelapenya Dec 4, 2024
f8d24c7
chore: deprecate ReadConfig in favour of NewConfig
mdelapenya Dec 5, 2024
786d79c
chore: line separator
mdelapenya Dec 5, 2024
4bd9934
chore: do not export var
mdelapenya Dec 5, 2024
5f7eb43
Merge branch 'main' into context-support
mdelapenya Dec 5, 2024
8fcd654
chore: bubble up error on environment configuration
mdelapenya Dec 5, 2024
13934da
chore: rename test cases
mdelapenya Dec 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docker_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -440,7 +440,7 @@ func TestContainerCreationWithName(t *testing.T) {
Name: creationName,
// no network means it will be connected to the default bridge network
// of any given container runtime
Networks: []string{},
// Networks: []string{},
},
Started: true,
})
Expand All @@ -456,7 +456,7 @@ func TestContainerCreationWithName(t *testing.T) {
require.NoError(t, err)
require.Lenf(t, networks, 1, "Expected networks 1. Got '%d'.", len(networks))
network := networks[0]
assert.Equalf(t, Bridge, network, "Expected network name '%s'. Got '%s'.", Bridge, network)
require.Equal(t, Bridge, network)

endpoint, err := nginxC.PortEndpoint(ctx, nginxDefaultPort, "http")
require.NoError(t, err)
Expand Down Expand Up @@ -671,7 +671,7 @@ func Test_BuildContainerFromDockerfileWithBuildLog(t *testing.T) {

temp := strings.Split(string(out), "\n")
require.NotEmpty(t, temp)
assert.Regexpf(t, `^(?i:Step)\s*1/\d+\s*:\s*FROM alpine$`, temp[0], "Expected stdout first line to be %s. Got '%s'.", "Step 1/* : FROM alpine", temp[0])
require.Regexp(t, `^(?i:Step)\s*1/\d+\s*:\s*FROM alpine$`, temp[0])
}

func TestContainerCreationWaitsForLogAndPortContextTimeout(t *testing.T) {
Expand Down
6 changes: 3 additions & 3 deletions docs/system_requirements/docker.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,10 +45,10 @@ docker context use orbstack

### Podman
mdelapenya marked this conversation as resolved.
Show resolved Hide resolved

Podman does not create its own Docker context when it is installed so, after starting a `podman-machine`, please create it with the following command:
Podman does not create its own Docker context when it is installed so, after starting a `podman-machine` in **rootful mode**, please create the context with the following command:

```sh
docker context create podman --description "podman context" --docker "host=unix:///var/folders/_j/nhbgdck523n3008dd3zlsm5m0000gn/T/podman/podman-machine-default-api.sock"
podman context create podman --description "podman context" --docker "host=unix:///var/folders/_j/nhbgdck523n3008dd3zlsm5m0000gn/T/podman/podman-machine-default-api.sock"
```

!!! note
Expand All @@ -57,7 +57,7 @@ docker context create podman --description "podman context" --docker "host=unix:
Then you can set this as the active context by running:

```sh
docker context use podman
podman context use podman
```

### Rancher Desktop
Expand Down
53 changes: 26 additions & 27 deletions internal/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,13 @@ import (
"github.com/magiconair/properties"
)

const ReaperDefaultImage = "testcontainers/ryuk:0.11.0"
const (
// ReaperDefaultImage is the default image used for Ryuk, the resource reaper.
ReaperDefaultImage = "testcontainers/ryuk:0.11.0"

// TestcontainersProperties is the name of the properties file used to configure Testcontainers.
TestcontainersProperties = ".testcontainers.properties"
mdelapenya marked this conversation as resolved.
Show resolved Hide resolved
)

var (
tcConfig Config
Expand Down Expand Up @@ -89,31 +95,6 @@ type Config struct {

// }

// defaultConfig
func defaultConfig() Config {
config := Config{}

home, err := os.UserHomeDir()
if err != nil {
return applyEnvironmentConfiguration(config)
}

tcProp := filepath.Join(home, ".testcontainers.properties")
// Init from a file, ignore if it doesn't exist, which is the case for most users.
// The properties library will return the default values for the struct.
props, err := properties.LoadFiles([]string{tcProp}, properties.UTF8, true)
if err != nil {
return applyEnvironmentConfiguration(config)
}

if err := props.Decode(&config); err != nil {
fmt.Printf("invalid testcontainers properties file, returning an empty Testcontainers configuration: %v\n", err)
return applyEnvironmentConfiguration(config)
}

return config
}

func applyEnvironmentConfiguration(config Config) Config {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug: callers can't distinguish actual success from hidden failure e.g. a bad boolean or duration.

ryukDisabledEnv := os.Getenv("TESTCONTAINERS_RYUK_DISABLED")
if parseBool(ryukDisabledEnv) {
Expand Down Expand Up @@ -167,7 +148,25 @@ func Reset() {
}

func read() Config {
config := defaultConfig()
var config Config

home, err := os.UserHomeDir()
if err != nil {
return applyEnvironmentConfiguration(config)
}

tcProp := filepath.Join(home, TestcontainersProperties)
// Init from a file, ignore if it doesn't exist, which is the case for most users.
// The properties library will return the default values for the struct.
props, err := properties.LoadFiles([]string{tcProp}, properties.UTF8, true)
if err != nil {
return applyEnvironmentConfiguration(config)
}

if err := props.Decode(&config); err != nil {
fmt.Printf("invalid testcontainers properties file, returning an empty Testcontainers configuration: %v\n", err)
mdelapenya marked this conversation as resolved.
Show resolved Hide resolved
return applyEnvironmentConfiguration(config)
}

return applyEnvironmentConfiguration(config)
stevenh marked this conversation as resolved.
Show resolved Hide resolved
}
Expand Down
15 changes: 13 additions & 2 deletions internal/config/config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,11 @@ func TestReadTCConfig(t *testing.T) {

config := read()

expected := defaultConfig()
// The time fields are set to the default values.
expected := Config{
RyukReconnectionTimeout: 10 * time.Second,
RyukConnectionTimeout: time.Minute,
}

assert.Equal(t, expected, config)
})
Expand All @@ -114,7 +118,14 @@ func TestReadTCConfig(t *testing.T) {
t.Setenv("DOCKER_HOST", tcpDockerHost33293)

config := read()
expected := defaultConfig() // the config does not read DOCKER_HOST, that's why it's empty

// The time fields are set to the default values,
// and the config does not read DOCKER_HOST,
// that's why it's empty
expected := Config{
RyukReconnectionTimeout: 10 * time.Second,
RyukConnectionTimeout: time.Minute,
}

assert.Equal(t, expected, config)
})
Expand Down
32 changes: 21 additions & 11 deletions internal/core/docker_context.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,23 +21,27 @@ import (
const (
// defaultContextName is the name reserved for the default context (config & env based)
defaultContextName = "default"

// envOverrideContext is the name of the environment variable that can be
mdelapenya marked this conversation as resolved.
Show resolved Hide resolved
// used to override the context to use. If set, it overrides the context
// that's set in the CLI's configuration file, but takes no effect if the
// "DOCKER_HOST" env-var is set (which takes precedence.
envOverrideContext = "DOCKER_CONTEXT"

// envOverrideConfigDir is the name of the environment variable that can be
// used to override the location of the client configuration files (~/.docker).
//
// It takes priority over the default.
envOverrideConfigDir = "DOCKER_CONFIG"

// configFileName is the name of the client configuration file inside the
// config-directory.
configFileName = "config.json"
configFileDir = ".docker"
contextsDir = "contexts"
metadataDir = "meta"
metaFile = "meta.json"

// DockerEndpoint is the name of the docker endpoint in a stored context
dockerEndpoint string = "docker"
)
Expand Down Expand Up @@ -118,6 +122,7 @@ type untypedContextMetadata struct {
Name string `json:"name,omitempty"`
}

// getByID returns the metadata for a context by its ID
func (s *metadataStore) getByID(id contextdir) (metadata, error) {
fileName := filepath.Join(s.contextDir(id), metaFile)
bytes, err := os.ReadFile(fileName)
Expand All @@ -134,17 +139,17 @@ func (s *metadataStore) getByID(id contextdir) (metadata, error) {
}

if err := json.Unmarshal(bytes, &untyped); err != nil {
return metadata{}, fmt.Errorf("parsing %s: %w", fileName, err)
return metadata{}, fmt.Errorf("parsing %s (metadata): %w", fileName, err)
}

r.Name = untyped.Name
if r.Metadata, err = parseTypedOrMap(untyped.Metadata, s.config.contextType); err != nil {
return metadata{}, fmt.Errorf("parsing %s: %w", fileName, err)
return metadata{}, fmt.Errorf("parsing %s (context type): %w", fileName, err)
}

for k, v := range untyped.Endpoints {
if r.Endpoints[k], err = parseTypedOrMap(v, s.config.endpointTypes[k]); err != nil {
return metadata{}, fmt.Errorf("parsing %s: %w", fileName, err)
return metadata{}, fmt.Errorf("parsing %s (endpoint types): %w", fileName, err)
}
}

Expand Down Expand Up @@ -176,8 +181,11 @@ func (s *metadataStore) list() ([]metadata, error) {
return res, nil
}

// contextdir is a type used to represent a context directory
type contextdir string

// isContextDir checks if the given path is a context directory,
// which means it contains a meta.json file.
func isContextDir(path string) bool {
s, err := os.Stat(filepath.Join(path, metaFile))
if err != nil {
Expand All @@ -187,33 +195,35 @@ func isContextDir(path string) bool {
return !s.IsDir()
}

// listRecursivelyMetadataDirs lists all directories that contain a meta.json file
func listRecursivelyMetadataDirs(root string) ([]string, error) {
fis, err := os.ReadDir(root)
fileEntries, err := os.ReadDir(root)
if err != nil {
return nil, err
}

var result []string
for _, fi := range fis {
if fi.IsDir() {
if isContextDir(filepath.Join(root, fi.Name())) {
result = append(result, fi.Name())
for _, fileEntry := range fileEntries {
if fileEntry.IsDir() {
if isContextDir(filepath.Join(root, fileEntry.Name())) {
result = append(result, fileEntry.Name())
}

subs, err := listRecursivelyMetadataDirs(filepath.Join(root, fi.Name()))
subs, err := listRecursivelyMetadataDirs(filepath.Join(root, fileEntry.Name()))
if err != nil {
return nil, err
}

for _, s := range subs {
result = append(result, filepath.Join(fi.Name(), s))
result = append(result, filepath.Join(fileEntry.Name(), s))
}
}
}

return result, nil
}

// parseTypedOrMap parses a JSON payload into a typed object or a map
func parseTypedOrMap(payload []byte, getter typeGetter) (any, error) {
if len(payload) == 0 || string(payload) == "null" {
return nil, nil
Expand Down Expand Up @@ -298,7 +308,7 @@ func GetDockerHostFromCurrentContext() (string, error) {
}
}

return "", ErrDockerSocketNotSetInDockerContext
return "", errDockerSocketNotSetInDockerContext
}

// currentContext returns the current context name, based on
Expand Down
48 changes: 25 additions & 23 deletions internal/core/docker_host.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,16 +19,16 @@
var DockerHostContextKey = dockerHostContext("docker_host")

var (
ErrDockerHostNotSet = errors.New("DOCKER_HOST is not set")
ErrDockerSocketOverrideNotSet = errors.New("TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE is not set")
ErrDockerSocketNotSetInContext = errors.New("socket not set in Go context")
ErrDockerSocketNotSetInDockerContext = errors.New("socket not set in Docker context")
ErrDockerSocketNotSetInProperties = errors.New("socket not set in ~/.testcontainers.properties")
ErrNoUnixSchema = errors.New("URL schema is not unix")
ErrSocketNotFound = errors.New("socket not found")
ErrSocketNotFoundInPath = errors.New("docker socket not found in " + DockerSocketPath)
// ErrTestcontainersHostNotSetInProperties this error is specific to Testcontainers
ErrTestcontainersHostNotSetInProperties = errors.New("tc.host not set in ~/.testcontainers.properties")
errDockerHostNotSet = errors.New("DOCKER_HOST is not set")
errDockerSocketOverrideNotSet = errors.New("TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE is not set")
errDockerSocketNotSetInContext = errors.New("socket not set in Go context")
errDockerSocketNotSetInDockerContext = errors.New("socket not set in Docker context")
errDockerSocketNotSetInProperties = errors.New("socket not set in ~/.testcontainers.properties")
errNoUnixSchema = errors.New("URL schema is not unix")
errSocketNotFound = errors.New("socket not found")
errSocketNotFoundInPath = errors.New("docker socket not found in " + DockerSocketPath)
// errTestcontainersHostNotSetInProperties this error is specific to Testcontainers
errTestcontainersHostNotSetInProperties = errors.New("tc.host not set in ~/.testcontainers.properties")
)

var (
Expand Down Expand Up @@ -127,7 +127,9 @@
testcontainersHostFromProperties,
dockerHostFromEnv,
dockerHostFromContext,
dockerHostFromDockerContext,
func(_ context.Context) (string, error) {
return GetDockerHostFromCurrentContext()
},
dockerSocketPath,
dockerHostFromProperties,
rootlessDockerSocketPath,
Expand Down Expand Up @@ -155,7 +157,7 @@
return "", errors.Join(errs...)
}

return "", ErrSocketNotFound
return "", errSocketNotFound
}

// extractDockerSocket Extracts the docker socket from the different alternatives, without caching the result.
Expand Down Expand Up @@ -232,11 +234,11 @@
// not being set, false otherwise.
func isHostNotSet(err error) bool {
switch {
case errors.Is(err, ErrTestcontainersHostNotSetInProperties),
errors.Is(err, ErrDockerHostNotSet),
errors.Is(err, ErrDockerSocketNotSetInContext),
errors.Is(err, ErrDockerSocketNotSetInProperties),
errors.Is(err, ErrSocketNotFoundInPath),
case errors.Is(err, errTestcontainersHostNotSetInProperties),
errors.Is(err, errDockerHostNotSet),
errors.Is(err, errDockerSocketNotSetInContext),
errors.Is(err, errDockerSocketNotSetInProperties),
errors.Is(err, errSocketNotFoundInPath),
errors.Is(err, ErrXDGRuntimeDirNotSet),
errors.Is(err, ErrRootlessDockerNotFoundHomeRunDir),
errors.Is(err, ErrRootlessDockerNotFoundHomeDesktopDir),
Expand All @@ -253,7 +255,7 @@
return dockerHostPath, nil
}

return "", ErrDockerHostNotSet
return "", errDockerHostNotSet
}

// dockerHostFromContext returns the docker host from the Go context, if it's not empty
Expand All @@ -267,11 +269,11 @@
return parsed, nil
}

return "", ErrDockerSocketNotSetInContext
return "", errDockerSocketNotSetInContext
}

// dockerHostFromContext returns the docker host from the Go context, if it's not empty
func dockerHostFromDockerContext(ctx context.Context) (string, error) {

Check failure on line 276 in internal/core/docker_host.go

View workflow job for this annotation

GitHub Actions / test (1.22.x, ubuntu-latest) / ./ubuntu-latest/1.22.x

func `dockerHostFromDockerContext` is unused (unused)

Check failure on line 276 in internal/core/docker_host.go

View workflow job for this annotation

GitHub Actions / Test with reaper off (1.22.x) / ./ubuntu-latest/1.22.x

func `dockerHostFromDockerContext` is unused (unused)

Check failure on line 276 in internal/core/docker_host.go

View workflow job for this annotation

GitHub Actions / Test with reaper off (1.x) / ./ubuntu-latest/1.x

func `dockerHostFromDockerContext` is unused (unused)

Check failure on line 276 in internal/core/docker_host.go

View workflow job for this annotation

GitHub Actions / Test with Rootless Docker (1.22.x, ubuntu-latest) / ./ubuntu-latest/1.22.x

func `dockerHostFromDockerContext` is unused (unused)

Check failure on line 276 in internal/core/docker_host.go

View workflow job for this annotation

GitHub Actions / Test with Rootless Docker (1.x, ubuntu-latest) / ./ubuntu-latest/1.x

func `dockerHostFromDockerContext` is unused (unused)
mdelapenya marked this conversation as resolved.
Show resolved Hide resolved
dockerHost, err := GetDockerHostFromCurrentContext()
if err == nil {
return dockerHost, nil
Expand All @@ -288,7 +290,7 @@
return socketPath, nil
}

return "", ErrDockerSocketNotSetInProperties
return "", errDockerSocketNotSetInProperties
}

// dockerSocketOverridePath returns the docker socket from the TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE environment variable,
Expand All @@ -298,7 +300,7 @@
return dockerHostPath, nil
}

return "", ErrDockerSocketOverrideNotSet
return "", errDockerSocketOverrideNotSet
}

// dockerSocketPath returns the docker socket from the default docker socket path, if it's not empty
Expand All @@ -308,7 +310,7 @@
return DockerSocketPathWithSchema, nil
}

return "", ErrSocketNotFoundInPath
return "", errSocketNotFoundInPath
}

// testcontainersHostFromProperties returns the testcontainers host from the ~/.testcontainers.properties file, if it's not empty
Expand All @@ -324,7 +326,7 @@
return parsed, nil
}

return "", ErrTestcontainersHostNotSetInProperties
return "", errTestcontainersHostNotSetInProperties
}

// InAContainer returns true if the code is running inside a container
Expand Down
Loading
Loading