Skip to content

Commit

Permalink
Deploying to gh-pages from @ 53a3579 🚀
Browse files Browse the repository at this point in the history
  • Loading branch information
d-rk committed Mar 8, 2024
1 parent 2fefab1 commit 00a5219
Show file tree
Hide file tree
Showing 114 changed files with 1,342 additions and 1,103 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ dist
kafkactl

### Configs
.kafkactl.yml
kafkactl.yml

### Snap
Expand Down
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## 5.0.1 - 2024-03-08

## 5.0.0 - 2024-03-08

### Added
- [#190](https://github.com/deviceinsight/kafkactl/pull/190) Improve handling of project config files
- [#192](https://github.com/deviceinsight/kafkactl/pull/192) Plugin infrastructure for tokenProviders

### Fixed
- [#151](https://github.com/deviceinsight/kafkactl/issues/151) Get topics command should not require describe permission

## 4.0.0 - 2024-01-18

### Added
Expand Down
97 changes: 91 additions & 6 deletions README.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -114,14 +114,21 @@ contexts:
kubeConfig: ~/.kube/config #optional
kubeContext: my-cluster
namespace: my-namespace
# optional: docker image to use (tag will be added by kafkactl based on the current version)
# optional: docker image to use (the tag of the image will be suffixed by `-scratch` or `-ubuntu` depending on command)
image: private.registry.com/deviceinsight/kafkactl
# optional: secret for private docker registry
imagePullSecret: registry-secret
# optional: extra args to be passed into the kubectl run command
extra:
- --overrides
- '{"spec": { "nodeSelector": {"kubernetes.io/hostname": "eks-staging-4"}}}'
# optional: serviceAccount to use for the pod
serviceAccount: my-service-account
# optional: labels to add to the pod
labels:
key: value
# optional: annotations to add to the pod
annotations:
key: value
# optional: nodeSelector to add to the pod
nodeSelector:
key: value
# optional: clientID config (defaults to kafkactl-{username})
clientID: my-client-id
Expand Down Expand Up @@ -163,21 +170,48 @@ contexts:
# optional: isolationLevel (defaults to ReadCommitted)
isolationLevel: ReadUncommitted
# optional for project config files
current-context: default
----

[#_config_file_read_order]
The config file location is resolved by

. checking for a provided commandline argument: `--config-file=$PATH_TO_CONFIG`
. evaluating the environment variable: `export KAFKA_CTL_CONFIG=$PATH_TO_CONFIG`
. checking for a config file in the working directory i.e. `$PWD/kafkactl.yml`
. checking for a project config file in the working directory (see <<_project_config_files>>)
. as default the config file is looked up from one of the following locations:
** `$HOME/.config/kafkactl/config.yml`
** `$HOME/.kafkactl/config.yml`
** `$SNAP_REAL_HOME/.kafkactl/config.yml`
** `$SNAP_DATA/kafkactl/config.yml`
** `/etc/kafkactl/config.yml`

[#_project_config_files]
==== Project config files

In addition to the config file locations above, _kafkactl_ allows to create a config file on project level.
A project config file is meant to be placed at the root level of a git repo and declares the kafka configuration
for this repository/project.

In order to identify the config file as belonging to _kafkactl_ the following names can be used:

* `kafkactl.yml`
* `.kafkactl.yml`

During initialization _kafkactl_ starts from the current working directory and recursively looks for a project level
config file. The recursive lookup ends at the boundary of a git repository (i.e. if a `.git` folder is found).
This way, _kafkactl_ can be used conveniently anywhere in the git repository.

Additionally, project config files have a special feature to use them read-only. Topically, if you configure more than
one context in a config file, and you switch the context with `kafkactl config use-context xy` this will lead to a write
operation on the config file to save the _current context_.

In order to avoid this for project config files, one can just omit the `current-context` parameter from the config file.
In this case _kafkactl_ will delegate read and write operations for the _current context_ to the next configuration file
according to <<_config_file_read_order, the config file read order>>.


=== Auto completion

==== bash
Expand Down Expand Up @@ -544,6 +578,57 @@ Producing protobuf message converted from JSON:
kafkactl produce my-topic --key='{"keyField":123}' --key-proto-type MyKeyMessage --value='{"valueField":"value"}' --value-proto-type MyValueMessage --proto-file kafkamsg.proto
----

A more complex protobuf message converted from a multi-line JSON string can be produced using a file input with custom separators.

For example, if you have the following protobuf definition (`complex.proto`):

[,protobuf]
----
syntax = "proto3";
import "google/protobuf/timestamp.proto";
message ComplexMessage {
CustomerInfo customer_info = 1;
DeviceInfo device_info = 2;
}
message CustomerInfo {
string customer_id = 1;
string name = 2;
}
message DeviceInfo {
string serial = 1;
google.protobuf.Timestamp last_update = 2;
}
----

And you have the following file (`complex-msg.txt`) that contains the key and value of the message:

[,text]
----
msg-key##
{
"customer_info": {
"customer_id": "12345",
"name": "Bob"
},
"device_info": {
"serial": "abcde",
"last_update": "2024-03-02T07:01:02.000Z"
}
}
+++
----

The command to produce the protobuf message using sample protobuf definition and input file would be:

[,bash]
----
kafkactl produce my-topic --value-proto-type=ComplexMessage --proto-file=complex.proto --lineSeparator='+++' --separator='##' --file=complex-msg.txt
----

=== Avro support

In order to enable avro support you just have to add the schema registry to your configuration:
Expand Down
14 changes: 7 additions & 7 deletions cmd/alter/alter-partition.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@ package alter
import (
"strconv"

"github.com/deviceinsight/kafkactl/cmd/validation"
"github.com/deviceinsight/kafkactl/internal/k8s"
"github.com/deviceinsight/kafkactl/internal/partition"
"github.com/deviceinsight/kafkactl/internal/topic"
"github.com/deviceinsight/kafkactl/output"
"github.com/deviceinsight/kafkactl/v5/cmd/validation"
"github.com/deviceinsight/kafkactl/v5/internal/k8s"
"github.com/deviceinsight/kafkactl/v5/internal/output"
"github.com/deviceinsight/kafkactl/v5/internal/partition"
"github.com/deviceinsight/kafkactl/v5/internal/topic"
"github.com/pkg/errors"
"github.com/spf13/cobra"
)
Expand Down Expand Up @@ -36,14 +36,14 @@ func newAlterPartitionCmd() *cobra.Command {
}
}
},
PreRunE: func(cmd *cobra.Command, args []string) error {
PreRunE: func(cmd *cobra.Command, _ []string) error {
return validation.ValidateAtLeastOneRequiredFlag(cmd)
},
ValidArgsFunction: func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
if len(args) == 0 {
return topic.CompleteTopicNames(cmd, args, toComplete)
} else if len(args) == 1 {
return partition.CompletePartitionIds(cmd, args, toComplete)
return partition.CompletePartitionIDs(cmd, args, toComplete)
}
return nil, cobra.ShellCompDirectiveNoFileComp
},
Expand Down
6 changes: 3 additions & 3 deletions cmd/alter/alter-partition_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ import (
"github.com/Rican7/retry"
"github.com/Rican7/retry/backoff"
"github.com/Rican7/retry/strategy"
"github.com/deviceinsight/kafkactl/internal/topic"
"github.com/deviceinsight/kafkactl/testutil"
"github.com/deviceinsight/kafkactl/v5/internal/testutil"
"github.com/deviceinsight/kafkactl/v5/internal/topic"
"gopkg.in/errgo.v2/fmt/errors"
)

Expand Down Expand Up @@ -61,7 +61,7 @@ func TestAlterPartitionReplicationFactorIntegration(t *testing.T) {
return
}

checkReplicas := func(attempt uint) error {
checkReplicas := func(_ uint) error {
_, err := kafkaCtl.Execute("describe", "topic", topicName, "-o", "yaml")

if err != nil {
Expand Down
10 changes: 5 additions & 5 deletions cmd/alter/alter-topic.go
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
package alter

import (
"github.com/deviceinsight/kafkactl/cmd/validation"
"github.com/deviceinsight/kafkactl/internal/k8s"
"github.com/deviceinsight/kafkactl/internal/topic"
"github.com/deviceinsight/kafkactl/output"
"github.com/deviceinsight/kafkactl/v5/cmd/validation"
"github.com/deviceinsight/kafkactl/v5/internal/k8s"
"github.com/deviceinsight/kafkactl/v5/internal/output"
"github.com/deviceinsight/kafkactl/v5/internal/topic"
"github.com/spf13/cobra"
)

Expand All @@ -23,7 +23,7 @@ func newAlterTopicCmd() *cobra.Command {
}
}
},
PreRunE: func(cmd *cobra.Command, args []string) error {
PreRunE: func(cmd *cobra.Command, _ []string) error {
return validation.ValidateAtLeastOneRequiredFlag(cmd)
},
ValidArgsFunction: topic.CompleteTopicNames,
Expand Down
10 changes: 5 additions & 5 deletions cmd/alter/alter-topic_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ import (
"github.com/Rican7/retry"
"github.com/Rican7/retry/backoff"
"github.com/Rican7/retry/strategy"
"github.com/deviceinsight/kafkactl/internal/topic"
"github.com/deviceinsight/kafkactl/testutil"
"github.com/deviceinsight/kafkactl/v5/internal/testutil"
"github.com/deviceinsight/kafkactl/v5/internal/topic"
"gopkg.in/errgo.v2/fmt/errors"
)

Expand Down Expand Up @@ -51,7 +51,7 @@ func TestAlterTopicPartitionsIntegration(t *testing.T) {
t.Fatalf("failed to execute command: %v", err)
}

getPartitions := func(attempt uint) error {
getPartitions := func(_ uint) error {
_, err := kafkaCtl.Execute("describe", "topic", topicName, "-o", "yaml")

if err != nil {
Expand Down Expand Up @@ -93,7 +93,7 @@ func TestAlterTopicIncreaseReplicationFactorIntegration(t *testing.T) {
return
}

checkReplicas := func(attempt uint) error {
checkReplicas := func(_ uint) error {
_, err := kafkaCtl.Execute("describe", "topic", topicName, "-o", "yaml")

if err != nil {
Expand Down Expand Up @@ -137,7 +137,7 @@ func TestAlterTopicDecreaseReplicationFactorIntegration(t *testing.T) {
return
}

checkReplicas := func(attempt uint) error {
checkReplicas := func(_ uint) error {
_, err := kafkaCtl.Execute("describe", "topic", topicName, "-o", "yaml")

if err != nil {
Expand Down
6 changes: 3 additions & 3 deletions cmd/attach/attach.go
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
package attach

import (
"github.com/deviceinsight/kafkactl/internal/k8s"
"github.com/deviceinsight/kafkactl/output"
"github.com/deviceinsight/kafkactl/v5/internal/k8s"
"github.com/deviceinsight/kafkactl/v5/internal/output"
"github.com/spf13/cobra"
)

Expand All @@ -12,7 +12,7 @@ func NewAttachCmd() *cobra.Command {
Use: "attach",
Short: "run kafkactl pod in kubernetes and attach to it",
Args: cobra.NoArgs,
Run: func(cobraCmd *cobra.Command, args []string) {
Run: func(_ *cobra.Command, _ []string) {
if err := k8s.NewOperation().Attach(); err != nil {
output.Fail(err)
}
Expand Down
8 changes: 4 additions & 4 deletions cmd/clone/clone-consumergroup.go
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
package clone

import (
"github.com/deviceinsight/kafkactl/internal/consumergroupoffsets"
"github.com/deviceinsight/kafkactl/internal/consumergroups"
"github.com/deviceinsight/kafkactl/internal/k8s"
"github.com/deviceinsight/kafkactl/output"
"github.com/deviceinsight/kafkactl/v5/internal/consumergroupoffsets"
"github.com/deviceinsight/kafkactl/v5/internal/consumergroups"
"github.com/deviceinsight/kafkactl/v5/internal/k8s"
"github.com/deviceinsight/kafkactl/v5/internal/output"
"github.com/spf13/cobra"
)

Expand Down
2 changes: 1 addition & 1 deletion cmd/clone/clone-consumergroup_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import (
"fmt"
"testing"

"github.com/deviceinsight/kafkactl/testutil"
"github.com/deviceinsight/kafkactl/v5/internal/testutil"
)

func TestCloneConsumerGroupIntegration(t *testing.T) {
Expand Down
6 changes: 3 additions & 3 deletions cmd/clone/clone-topic.go
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
package clone

import (
"github.com/deviceinsight/kafkactl/internal/k8s"
"github.com/deviceinsight/kafkactl/internal/topic"
"github.com/deviceinsight/kafkactl/output"
"github.com/deviceinsight/kafkactl/v5/internal/k8s"
"github.com/deviceinsight/kafkactl/v5/internal/output"
"github.com/deviceinsight/kafkactl/v5/internal/topic"
"github.com/spf13/cobra"
)

Expand Down
6 changes: 3 additions & 3 deletions cmd/clone/clone-topic_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ import (
"github.com/Rican7/retry/strategy"
"gopkg.in/errgo.v2/fmt/errors"

"github.com/deviceinsight/kafkactl/internal/topic"
"github.com/deviceinsight/kafkactl/testutil"
"github.com/deviceinsight/kafkactl/v5/internal/testutil"
"github.com/deviceinsight/kafkactl/v5/internal/topic"
)

func TestCloneTopicIntegration(t *testing.T) {
Expand All @@ -29,7 +29,7 @@ func TestCloneTopicIntegration(t *testing.T) {

testutil.AssertEquals(t, fmt.Sprintf("topic %s cloned to %s", srcTopic, targetTopic), kafkaCtl.GetStdOut())

getTopic := func(attempt uint) error {
getTopic := func(_ uint) error {
_, err := kafkaCtl.Execute("describe", "topic", targetTopic, "-o", "yaml")

if err != nil {
Expand Down
2 changes: 1 addition & 1 deletion cmd/completion.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ package cmd
import (
"os"

"github.com/deviceinsight/kafkactl/output"
"github.com/deviceinsight/kafkactl/v5/internal/output"
"github.com/pkg/errors"
"github.com/spf13/cobra"
)
Expand Down
8 changes: 4 additions & 4 deletions cmd/config/currentContext.go
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
package config

import (
"github.com/deviceinsight/kafkactl/output"
"github.com/deviceinsight/kafkactl/v5/internal/global"
"github.com/deviceinsight/kafkactl/v5/internal/output"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)

func newCurrentContextCmd() *cobra.Command {
Expand All @@ -12,8 +12,8 @@ func newCurrentContextCmd() *cobra.Command {
Aliases: []string{"currentContext"},
Short: "show current context",
Long: `Displays the name of context that is currently active`,
Run: func(cmd *cobra.Command, args []string) {
context := viper.GetString("current-context")
Run: func(_ *cobra.Command, _ []string) {
context := global.GetCurrentContext()
output.Infof("%s", context)
},
}
Expand Down
Loading

0 comments on commit 00a5219

Please sign in to comment.