diff --git a/docs/_static/cmd-gazctl-attach-uuids.txt b/docs/_static/cmd-gazctl-attach-uuids.txt index 775a1dce..ebeeeadd 100644 --- a/docs/_static/cmd-gazctl-attach-uuids.txt +++ b/docs/_static/cmd-gazctl-attach-uuids.txt @@ -42,18 +42,11 @@ Optionally compose with "jq" to un-nest the JSON objects: > | jq -c '{uuid: .uuid} + .record' -Application Options: - --zone= Availability zone within which this process is running (default: local) [$ZONE] - -Logging: - --log.level=[info|debug|warn] Logging level (default: info) [$LOG_LEVEL] - --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] - Help Options: - -h, --help Show this help message + -h, --help Show this help message [attach-uuids command options] - --template= Go text/template for output (default: "{{.UUID}},{{.Line}}\n") - --max-length= Maximum allowed byte-length of an input line (default: 4194304) - --skip-header Omit the first line of each input file + --template= Go text/template for output (default: "{{.UUID}},{{.Line}}\n") + --max-length= Maximum allowed byte-length of an input line (default: 4194304) + --skip-header Omit the first line of each input file diff --git a/docs/_static/cmd-gazctl-journals-append.txt b/docs/_static/cmd-gazctl-journals-append.txt index 84eccdf2..2b191b1e 100644 --- a/docs/_static/cmd-gazctl-journals-append.txt +++ b/docs/_static/cmd-gazctl-journals-append.txt @@ -53,27 +53,34 @@ cat /dev/stdout > my-fifo & # Hold my-fifo open so gazctl doesn't read EOF. gazctl journals append -l my-label --framing 'lines' --mapping 'rendezvous' --input my-fifo -Application Options: - --zone= Availability zone within which this process is running (default: local) [$ZONE] - -Logging: - --log.level=[info|debug|warn] Logging level (default: info) [$LOG_LEVEL] - --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] - Help Options: - -h, --help Show this help message + -h, --help Show this help message [journals command options] + Interact with broker journals: + --zone= Availability zone within which this process is running (default: local) [$ZONE] + + Logging: + --log.level=[trace|debug|info|warn|error|fatal] Logging level (default: warn) [$LOG_LEVEL] + --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] + Broker: - --broker.address= Service address endpoint (default: http://localhost:8080) [$BROKER_ADDRESS] - --broker.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) (default: 0) [$BROKER_CACHE_SIZE] - --broker.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$BROKER_CACHE_TTL] + --broker.address= Service address endpoint (default: http://localhost:8080) [$BROKER_ADDRESS] + --broker.cert-file= Path to the client TLS certificate [$BROKER_CERT_FILE] + --broker.cert-key-file= Path to the client TLS private key [$BROKER_CERT_KEY_FILE] + --broker.trusted-ca-file= Path to the trusted CA for client verification of server certificates + [$BROKER_TRUSTED_CA_FILE] + --broker.auth-keys= Whitespace or comma separated, base64-encoded keys. The first key is used to sign + Authorization tokens. [$BROKER_AUTH_KEYS] + --broker.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) + (default: 0) [$BROKER_CACHE_SIZE] + --broker.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$BROKER_CACHE_TTL] [append command options] - -l, --selector= Label selector of journals to append to - -i, --input= Input file path. Use '-' for stdin (default: -) - -f, --framing=[none|lines|fixed] Framing of records in input, if any (default: none) - -m, --mapping=[random|modulo|rendezvous|direct] Mapping function of records to journals (default: random) - --base64 Partition keys under 'lines' framing are interpreted as base64 + -l, --selector= Label selector of journals to append to + -i, --input= Input file path. Use '-' for stdin (default: -) + -f, --framing=[none|lines|fixed] Framing of records in input, if any (default: none) + -m, --mapping=[random|modulo|rendezvous|direct] Mapping function of records to journals (default: random) + --base64 Partition keys under 'lines' framing are interpreted as base64 diff --git a/docs/_static/cmd-gazctl-journals-apply.txt b/docs/_static/cmd-gazctl-journals-apply.txt index d81de908..bd940693 100644 --- a/docs/_static/cmd-gazctl-journals-apply.txt +++ b/docs/_static/cmd-gazctl-journals-apply.txt @@ -35,26 +35,33 @@ gazctl supports a flag which will send changes in batches of at most a single Etcd transaction and it should therefore be used with caution. If possible, prefer to use label selectors to limit the number of changes. -Application Options: - --zone= Availability zone within which this process is running (default: local) [$ZONE] - -Logging: - --log.level=[info|debug|warn] Logging level (default: info) [$LOG_LEVEL] - --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] - Help Options: - -h, --help Show this help message + -h, --help Show this help message [journals command options] + Interact with broker journals: + --zone= Availability zone within which this process is running (default: local) [$ZONE] + + Logging: + --log.level=[trace|debug|info|warn|error|fatal] Logging level (default: warn) [$LOG_LEVEL] + --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] + Broker: - --broker.address= Service address endpoint (default: http://localhost:8080) [$BROKER_ADDRESS] - --broker.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) (default: 0) [$BROKER_CACHE_SIZE] - --broker.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$BROKER_CACHE_TTL] + --broker.address= Service address endpoint (default: http://localhost:8080) [$BROKER_ADDRESS] + --broker.cert-file= Path to the client TLS certificate [$BROKER_CERT_FILE] + --broker.cert-key-file= Path to the client TLS private key [$BROKER_CERT_KEY_FILE] + --broker.trusted-ca-file= Path to the trusted CA for client verification of server certificates + [$BROKER_TRUSTED_CA_FILE] + --broker.auth-keys= Whitespace or comma separated, base64-encoded keys. The first key is used to sign + Authorization tokens. [$BROKER_AUTH_KEYS] + --broker.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) + (default: 0) [$BROKER_CACHE_SIZE] + --broker.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$BROKER_CACHE_TTL] [apply command options] - --specs= Input specifications path to apply. Use '-' for stdin (default: -) - --dry-run Perform a dry-run of the apply - --max-txn-size= maximum number of specs to be processed within an apply transaction. If 0, the default, all changes are issued in a single transaction - (default: 0) + --specs= Input specifications path to apply. Use '-' for stdin (default: -) + --dry-run Perform a dry-run of the apply + --max-txn-size= maximum number of specs to be processed within an apply transaction. If 0, the default, + all changes are issued in a single transaction (default: 0) diff --git a/docs/_static/cmd-gazctl-journals-edit.txt b/docs/_static/cmd-gazctl-journals-edit.txt index 130b1b94..5e3d6e6e 100644 --- a/docs/_static/cmd-gazctl-journals-edit.txt +++ b/docs/_static/cmd-gazctl-journals-edit.txt @@ -29,25 +29,32 @@ gazctl supports a flag which will send changes in batches of at most a single Etcd transaction and it should therefore be used with caution. If possible, prefer to use label selectors to limit the number of changes. -Application Options: - --zone= Availability zone within which this process is running (default: local) [$ZONE] - -Logging: - --log.level=[info|debug|warn] Logging level (default: info) [$LOG_LEVEL] - --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] - Help Options: - -h, --help Show this help message + -h, --help Show this help message [journals command options] + Interact with broker journals: + --zone= Availability zone within which this process is running (default: local) [$ZONE] + + Logging: + --log.level=[trace|debug|info|warn|error|fatal] Logging level (default: warn) [$LOG_LEVEL] + --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] + Broker: - --broker.address= Service address endpoint (default: http://localhost:8080) [$BROKER_ADDRESS] - --broker.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) (default: 0) [$BROKER_CACHE_SIZE] - --broker.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$BROKER_CACHE_TTL] + --broker.address= Service address endpoint (default: http://localhost:8080) [$BROKER_ADDRESS] + --broker.cert-file= Path to the client TLS certificate [$BROKER_CERT_FILE] + --broker.cert-key-file= Path to the client TLS private key [$BROKER_CERT_KEY_FILE] + --broker.trusted-ca-file= Path to the trusted CA for client verification of server certificates + [$BROKER_TRUSTED_CA_FILE] + --broker.auth-keys= Whitespace or comma separated, base64-encoded keys. The first key is used to sign + Authorization tokens. [$BROKER_AUTH_KEYS] + --broker.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) + (default: 0) [$BROKER_CACHE_SIZE] + --broker.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$BROKER_CACHE_TTL] [edit command options] - -l, --selector= Label Selector query to filter on - --max-txn-size= maximum number of specs to be processed within an apply transaction. If 0, the default, all changes are issued in a single transaction - (default: 0) + -l, --selector= Label Selector query to filter on + --max-txn-size= maximum number of specs to be processed within an apply transaction. If 0, the default, + all changes are issued in a single transaction (default: 0) diff --git a/docs/_static/cmd-gazctl-journals-fragments.txt b/docs/_static/cmd-gazctl-journals-fragments.txt index 846e4305..388e8337 100644 --- a/docs/_static/cmd-gazctl-journals-fragments.txt +++ b/docs/_static/cmd-gazctl-journals-fragments.txt @@ -51,27 +51,34 @@ gazctl journals fragments -l name=my/journal --url-ttl 1m --from $(date -d "1 ho gazctl journals fragments -l my-label --format json --url-ttl 1h --from $(date -d 3AM '+%s') --to $(date -d 4:05AM '+%s') --format json -Application Options: - --zone= Availability zone within which this process is running (default: local) [$ZONE] - -Logging: - --log.level=[info|debug|warn] Logging level (default: info) [$LOG_LEVEL] - --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] - Help Options: - -h, --help Show this help message + -h, --help Show this help message [journals command options] + Interact with broker journals: + --zone= Availability zone within which this process is running (default: local) [$ZONE] + + Logging: + --log.level=[trace|debug|info|warn|error|fatal] Logging level (default: warn) [$LOG_LEVEL] + --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] + Broker: - --broker.address= Service address endpoint (default: http://localhost:8080) [$BROKER_ADDRESS] - --broker.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) (default: 0) [$BROKER_CACHE_SIZE] - --broker.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$BROKER_CACHE_TTL] + --broker.address= Service address endpoint (default: http://localhost:8080) [$BROKER_ADDRESS] + --broker.cert-file= Path to the client TLS certificate [$BROKER_CERT_FILE] + --broker.cert-key-file= Path to the client TLS private key [$BROKER_CERT_KEY_FILE] + --broker.trusted-ca-file= Path to the trusted CA for client verification of server certificates + [$BROKER_TRUSTED_CA_FILE] + --broker.auth-keys= Whitespace or comma separated, base64-encoded keys. The first key is used to sign + Authorization tokens. [$BROKER_AUTH_KEYS] + --broker.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) + (default: 0) [$BROKER_CACHE_SIZE] + --broker.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$BROKER_CACHE_TTL] [fragments command options] - -l, --selector= Label Selector query to filter on - -o, --format=[table|json|proto] Output format (default: table) - --from= Restrict to fragments created at or after this time, in unix seconds since epoch - --to= Restrict to fragments created before this time, in unix seconds since epoch - --url-ttl= Provide a signed GET URL with the given TTL + -l, --selector= Label Selector query to filter on + -o, --format=[table|json|proto] Output format (default: table) + --from= Restrict to fragments created at or after this time, in unix seconds since epoch + --to= Restrict to fragments created before this time, in unix seconds since epoch + --url-ttl= Provide a signed GET URL with the given TTL (default: 0s) diff --git a/docs/_static/cmd-gazctl-journals-list.txt b/docs/_static/cmd-gazctl-journals-list.txt index 5b23e646..4ed065ed 100644 --- a/docs/_static/cmd-gazctl-journals-list.txt +++ b/docs/_static/cmd-gazctl-journals-list.txt @@ -24,29 +24,37 @@ typically, common configuration. This hierarchy is simply sugar for and is exactly equivalent to the original JournalSpecs. -Application Options: - --zone= Availability zone within which this process is running (default: local) [$ZONE] - -Logging: - --log.level=[info|debug|warn] Logging level (default: info) [$LOG_LEVEL] - --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] - Help Options: - -h, --help Show this help message + -h, --help Show this help message [journals command options] + Interact with broker journals: + --zone= Availability zone within which this process is running (default: local) [$ZONE] + + Logging: + --log.level=[trace|debug|info|warn|error|fatal] Logging level (default: warn) [$LOG_LEVEL] + --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] + Broker: - --broker.address= Service address endpoint (default: http://localhost:8080) [$BROKER_ADDRESS] - --broker.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) (default: 0) [$BROKER_CACHE_SIZE] - --broker.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$BROKER_CACHE_TTL] + --broker.address= Service address endpoint (default: http://localhost:8080) [$BROKER_ADDRESS] + --broker.cert-file= Path to the client TLS certificate [$BROKER_CERT_FILE] + --broker.cert-key-file= Path to the client TLS private key [$BROKER_CERT_KEY_FILE] + --broker.trusted-ca-file= Path to the trusted CA for client verification of server certificates + [$BROKER_TRUSTED_CA_FILE] + --broker.auth-keys= Whitespace or comma separated, base64-encoded keys. The first key is used to sign + Authorization tokens. [$BROKER_AUTH_KEYS] + --broker.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) + (default: 0) [$BROKER_CACHE_SIZE] + --broker.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$BROKER_CACHE_TTL] [list command options] - -l, --selector= Label Selector query to filter on - -o, --format=[table|yaml|json|proto] Output format (default: table) - -L, --label-columns= Labels to present as columns, eg -L label-one -L label-two - -p, --primary Show primary column - -r, --replicas Show replicas column - --rf Show replication factor column - --stores Show fragment store column + -l, --selector= Label Selector query to filter on + -o, --format=[table|yaml|json|proto] Output format (default: table) + -L, --label-columns= Labels to present as columns, eg -L label-one -L label-two + -p, --primary Show primary column + -r, --replicas Show replicas column + --rf Show replication factor column + --stores Show fragment store column + --watch Use a long-lived watch diff --git a/docs/_static/cmd-gazctl-journals-prune.txt b/docs/_static/cmd-gazctl-journals-prune.txt index ff7f352e..bb3eabdd 100644 --- a/docs/_static/cmd-gazctl-journals-prune.txt +++ b/docs/_static/cmd-gazctl-journals-prune.txt @@ -3,32 +3,40 @@ Usage: Deletes fragments across all configured fragment stores of matching journals that are older than the configured retention. -There is a caveat when pruning journals. For a given journal, there could be multiple fragments covering the same offset. These fragments contain identical data at a given -offset, but the brokers are tracking only the largest fragment, i.e. the fragment that covers the largest span of offsets. As a result, the prune command will delete only this -tracked fragment, leaving the smaller fragments untouched. As a workaround, operators can wait for the fragment listing to refresh and prune the journals again. +There is a caveat when pruning journals. For a given journal, there could be multiple fragments covering the same offset. These fragments contain +identical data at a given offset, but the brokers are tracking only the largest fragment, i.e. the fragment that covers the largest span of +offsets. As a result, the prune command will delete only this tracked fragment, leaving the smaller fragments untouched. As a workaround, +operators can wait for the fragment listing to refresh and prune the journals again. Use --selector to supply a LabelSelector to select journals to prune. See "journals list --help" for details and examples. -Application Options: - --zone= Availability zone within which this process is running (default: local) [$ZONE] - -Logging: - --log.level=[info|debug|warn] Logging level (default: info) [$LOG_LEVEL] - --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] - Help Options: - -h, --help Show this help message + -h, --help Show this help message [journals command options] + Interact with broker journals: + --zone= Availability zone within which this process is running (default: local) [$ZONE] + + Logging: + --log.level=[trace|debug|info|warn|error|fatal] Logging level (default: warn) [$LOG_LEVEL] + --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] + Broker: - --broker.address= Service address endpoint (default: http://localhost:8080) [$BROKER_ADDRESS] - --broker.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) (default: 0) [$BROKER_CACHE_SIZE] - --broker.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$BROKER_CACHE_TTL] + --broker.address= Service address endpoint (default: http://localhost:8080) [$BROKER_ADDRESS] + --broker.cert-file= Path to the client TLS certificate [$BROKER_CERT_FILE] + --broker.cert-key-file= Path to the client TLS private key [$BROKER_CERT_KEY_FILE] + --broker.trusted-ca-file= Path to the trusted CA for client verification of server certificates + [$BROKER_TRUSTED_CA_FILE] + --broker.auth-keys= Whitespace or comma separated, base64-encoded keys. The first key is used to sign + Authorization tokens. [$BROKER_AUTH_KEYS] + --broker.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) + (default: 0) [$BROKER_CACHE_SIZE] + --broker.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$BROKER_CACHE_TTL] [prune command options] - -l, --selector= Label Selector query to filter on - --dry-run Perform a dry-run of the apply + -l, --selector= Label Selector query to filter on + --dry-run Perform a dry-run of the apply diff --git a/docs/_static/cmd-gazctl-journals-read.txt b/docs/_static/cmd-gazctl-journals-read.txt index 1affd294..a0f0fdbd 100644 --- a/docs/_static/cmd-gazctl-journals-read.txt +++ b/docs/_static/cmd-gazctl-journals-read.txt @@ -58,29 +58,37 @@ echo "{}" > offsets.json # Must already exist. gazctl journals read -l my-label -o output --offsets offsets.json --offsets-out offsets.json --broker.cache.size=256 --zone=us-east-1 -Application Options: - --zone= Availability zone within which this process is running (default: local) [$ZONE] - -Logging: - --log.level=[info|debug|warn] Logging level (default: info) [$LOG_LEVEL] - --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] - Help Options: - -h, --help Show this help message + -h, --help Show this help message [journals command options] + Interact with broker journals: + --zone= Availability zone within which this process is running (default: local) [$ZONE] + + Logging: + --log.level=[trace|debug|info|warn|error|fatal] Logging level (default: warn) [$LOG_LEVEL] + --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] + Broker: - --broker.address= Service address endpoint (default: http://localhost:8080) [$BROKER_ADDRESS] - --broker.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) (default: 0) [$BROKER_CACHE_SIZE] - --broker.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$BROKER_CACHE_TTL] + --broker.address= Service address endpoint (default: http://localhost:8080) [$BROKER_ADDRESS] + --broker.cert-file= Path to the client TLS certificate [$BROKER_CERT_FILE] + --broker.cert-key-file= Path to the client TLS private key [$BROKER_CERT_KEY_FILE] + --broker.trusted-ca-file= Path to the trusted CA for client verification of server certificates + [$BROKER_TRUSTED_CA_FILE] + --broker.auth-keys= Whitespace or comma separated, base64-encoded keys. The first key is used to sign + Authorization tokens. [$BROKER_AUTH_KEYS] + --broker.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) + (default: 0) [$BROKER_CACHE_SIZE] + --broker.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$BROKER_CACHE_TTL] [read command options] - -l, --selector= Label selector of journals to read - -b, --block Do not exit on journal EOF; wait for new data until signaled - --tail Start reading from the journal write-head (rather than offset 0) - -o, --output= Output file path. Use '-' for stdout (default: -) - --offsets= Path from which initial journal offsets are read at startup - --offsets-out= Path to which final journal offsets are written at exit - --file-root= Filesystem path which roots file:// fragment store + -l, --selector= Label selector of journals to read + -b, --block Do not exit on journal EOF; wait for new data until signaled + --tail Start reading from the journal write-head (rather than offset 0) + -o, --output= Output file path. Use '-' for stdout (default: -) + --offsets= Path from which initial journal offsets are read at startup + --offsets-out= Path to which final journal offsets are written at exit + --file-root= Filesystem path which roots file:// fragment store + --from= Skip over fragments persisted before this time, in unix seconds since epoch diff --git a/docs/_static/cmd-gazctl-journals-reset-head.txt b/docs/_static/cmd-gazctl-journals-reset-head.txt index ca11d1d3..70672488 100644 --- a/docs/_static/cmd-gazctl-journals-reset-head.txt +++ b/docs/_static/cmd-gazctl-journals-reset-head.txt @@ -37,23 +37,30 @@ reset-head is safe to run against journals which are already consistent and and are being actively appended to. -Application Options: - --zone= Availability zone within which this process is running (default: local) [$ZONE] - -Logging: - --log.level=[info|debug|warn] Logging level (default: info) [$LOG_LEVEL] - --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] - Help Options: - -h, --help Show this help message + -h, --help Show this help message [journals command options] + Interact with broker journals: + --zone= Availability zone within which this process is running (default: local) [$ZONE] + + Logging: + --log.level=[trace|debug|info|warn|error|fatal] Logging level (default: warn) [$LOG_LEVEL] + --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] + Broker: - --broker.address= Service address endpoint (default: http://localhost:8080) [$BROKER_ADDRESS] - --broker.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) (default: 0) [$BROKER_CACHE_SIZE] - --broker.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$BROKER_CACHE_TTL] + --broker.address= Service address endpoint (default: http://localhost:8080) [$BROKER_ADDRESS] + --broker.cert-file= Path to the client TLS certificate [$BROKER_CERT_FILE] + --broker.cert-key-file= Path to the client TLS private key [$BROKER_CERT_KEY_FILE] + --broker.trusted-ca-file= Path to the trusted CA for client verification of server certificates + [$BROKER_TRUSTED_CA_FILE] + --broker.auth-keys= Whitespace or comma separated, base64-encoded keys. The first key is used to sign + Authorization tokens. [$BROKER_AUTH_KEYS] + --broker.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) + (default: 0) [$BROKER_CACHE_SIZE] + --broker.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$BROKER_CACHE_TTL] [reset-head command options] - -l, --selector= Label Selector query to filter on + -l, --selector= Label Selector query to filter on diff --git a/docs/_static/cmd-gazctl-journals-suspend.txt b/docs/_static/cmd-gazctl-journals-suspend.txt new file mode 100644 index 00000000..c5169815 --- /dev/null +++ b/docs/_static/cmd-gazctl-journals-suspend.txt @@ -0,0 +1,45 @@ +Usage: + gazctl [OPTIONS] journals [journals-OPTIONS] suspend [suspend-OPTIONS] + +Suspend idle journals to use fewer or zero broker replicas. + +Suspension updates the 'suspend' field of the JournalSpec with its suspended +status and its resumption offset. When applying other updates to JournalSpecs, +operators utilizing journal suspension must take care to not overwrite the +journal's suspension configuration. + +Typically this means reading the current JournalSpec and its ModRevision, +copying the current 'suspend' field alongside other changes being made, +and then applying the updated JournalSpec with ExpectModRevision. + +The 'journals edit' subcommand uses this workflow and is safe to use with suspension. + + +Help Options: + -h, --help Show this help message + +[journals command options] + + Interact with broker journals: + --zone= Availability zone within which this process is running (default: local) [$ZONE] + + Logging: + --log.level=[trace|debug|info|warn|error|fatal] Logging level (default: warn) [$LOG_LEVEL] + --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] + + Broker: + --broker.address= Service address endpoint (default: http://localhost:8080) [$BROKER_ADDRESS] + --broker.cert-file= Path to the client TLS certificate [$BROKER_CERT_FILE] + --broker.cert-key-file= Path to the client TLS private key [$BROKER_CERT_KEY_FILE] + --broker.trusted-ca-file= Path to the trusted CA for client verification of server certificates + [$BROKER_TRUSTED_CA_FILE] + --broker.auth-keys= Whitespace or comma separated, base64-encoded keys. The first key is used to sign + Authorization tokens. [$BROKER_AUTH_KEYS] + --broker.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) + (default: 0) [$BROKER_CACHE_SIZE] + --broker.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$BROKER_CACHE_TTL] + +[suspend command options] + -l, --selector= Label Selector query to filter on + -f, --force Suspend journals even if they have local fragments + diff --git a/docs/_static/cmd-gazctl-print-config.txt b/docs/_static/cmd-gazctl-print-config.txt index 024a846f..4c798129 100644 --- a/docs/_static/cmd-gazctl-print-config.txt +++ b/docs/_static/cmd-gazctl-print-config.txt @@ -5,13 +5,6 @@ print-config parses the combined configuration from gazctl.ini, flags, and environment variables, and then writes the configuration to stdout in INI format. -Application Options: - --zone= Availability zone within which this process is running (default: local) [$ZONE] - -Logging: - --log.level=[info|debug|warn] Logging level (default: info) [$LOG_LEVEL] - --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] - Help Options: - -h, --help Show this help message + -h, --help Show this help message diff --git a/docs/_static/cmd-gazctl-shards-apply.txt b/docs/_static/cmd-gazctl-shards-apply.txt index 65398904..092adc0f 100644 --- a/docs/_static/cmd-gazctl-shards-apply.txt +++ b/docs/_static/cmd-gazctl-shards-apply.txt @@ -25,31 +25,45 @@ gazctl supports a flag which will send changes in batches of at most a single Etcd transaction and it should therefore be used with caution. If possible, prefer to use label selectors to limit the number of changes. -Application Options: - --zone= Availability zone within which this process is running (default: local) [$ZONE] - -Logging: - --log.level=[info|debug|warn] Logging level (default: info) [$LOG_LEVEL] - --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] - Help Options: - -h, --help Show this help message + -h, --help Show this help message [shards command options] + Interact with consumer shards: + --zone= Availability zone within which this process is running (default: local) [$ZONE] + + Logging: + --log.level=[trace|debug|info|warn|error|fatal] Logging level (default: warn) [$LOG_LEVEL] + --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] + Consumer: - --consumer.address= Service address endpoint (default: http://localhost:8080) [$CONSUMER_ADDRESS] - --consumer.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) (default: 0) [$CONSUMER_CACHE_SIZE] - --consumer.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$CONSUMER_CACHE_TTL] + --consumer.address= Service address endpoint (default: http://localhost:8080) [$CONSUMER_ADDRESS] + --consumer.cert-file= Path to the client TLS certificate [$CONSUMER_CERT_FILE] + --consumer.cert-key-file= Path to the client TLS private key [$CONSUMER_CERT_KEY_FILE] + --consumer.trusted-ca-file= Path to the trusted CA for client verification of server certificates + [$CONSUMER_TRUSTED_CA_FILE] + --consumer.auth-keys= Whitespace or comma separated, base64-encoded keys. The first key is used to sign + Authorization tokens. [$CONSUMER_AUTH_KEYS] + --consumer.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) + (default: 0) [$CONSUMER_CACHE_SIZE] + --consumer.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$CONSUMER_CACHE_TTL] Broker: - --broker.address= Service address endpoint (default: http://localhost:8080) [$BROKER_ADDRESS] - --broker.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) (default: 0) [$BROKER_CACHE_SIZE] - --broker.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$BROKER_CACHE_TTL] + --broker.address= Service address endpoint (default: http://localhost:8080) [$BROKER_ADDRESS] + --broker.cert-file= Path to the client TLS certificate [$BROKER_CERT_FILE] + --broker.cert-key-file= Path to the client TLS private key [$BROKER_CERT_KEY_FILE] + --broker.trusted-ca-file= Path to the trusted CA for client verification of server certificates + [$BROKER_TRUSTED_CA_FILE] + --broker.auth-keys= Whitespace or comma separated, base64-encoded keys. The first key is used to sign + Authorization tokens. [$BROKER_AUTH_KEYS] + --broker.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) + (default: 0) [$BROKER_CACHE_SIZE] + --broker.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$BROKER_CACHE_TTL] [apply command options] - --specs= Input specifications path to apply. Use '-' for stdin (default: -) - --dry-run Perform a dry-run of the apply - --max-txn-size= maximum number of specs to be processed within an apply transaction. If 0, the default, all changes are issued in a single transaction - (default: 0) + --specs= Input specifications path to apply. Use '-' for stdin (default: -) + --dry-run Perform a dry-run of the apply + --max-txn-size= maximum number of specs to be processed within an apply transaction. If 0, the default, + all changes are issued in a single transaction (default: 0) diff --git a/docs/_static/cmd-gazctl-shards-edit.txt b/docs/_static/cmd-gazctl-shards-edit.txt index 057c37bf..d37fff1b 100644 --- a/docs/_static/cmd-gazctl-shards-edit.txt +++ b/docs/_static/cmd-gazctl-shards-edit.txt @@ -29,30 +29,44 @@ gazctl supports a flag which will send changes in batches of at most a single Etcd transaction and it should therefore be used with caution. If possible, prefer to use label selectors to limit the number of changes. -Application Options: - --zone= Availability zone within which this process is running (default: local) [$ZONE] - -Logging: - --log.level=[info|debug|warn] Logging level (default: info) [$LOG_LEVEL] - --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] - Help Options: - -h, --help Show this help message + -h, --help Show this help message [shards command options] + Interact with consumer shards: + --zone= Availability zone within which this process is running (default: local) [$ZONE] + + Logging: + --log.level=[trace|debug|info|warn|error|fatal] Logging level (default: warn) [$LOG_LEVEL] + --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] + Consumer: - --consumer.address= Service address endpoint (default: http://localhost:8080) [$CONSUMER_ADDRESS] - --consumer.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) (default: 0) [$CONSUMER_CACHE_SIZE] - --consumer.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$CONSUMER_CACHE_TTL] + --consumer.address= Service address endpoint (default: http://localhost:8080) [$CONSUMER_ADDRESS] + --consumer.cert-file= Path to the client TLS certificate [$CONSUMER_CERT_FILE] + --consumer.cert-key-file= Path to the client TLS private key [$CONSUMER_CERT_KEY_FILE] + --consumer.trusted-ca-file= Path to the trusted CA for client verification of server certificates + [$CONSUMER_TRUSTED_CA_FILE] + --consumer.auth-keys= Whitespace or comma separated, base64-encoded keys. The first key is used to sign + Authorization tokens. [$CONSUMER_AUTH_KEYS] + --consumer.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) + (default: 0) [$CONSUMER_CACHE_SIZE] + --consumer.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$CONSUMER_CACHE_TTL] Broker: - --broker.address= Service address endpoint (default: http://localhost:8080) [$BROKER_ADDRESS] - --broker.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) (default: 0) [$BROKER_CACHE_SIZE] - --broker.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$BROKER_CACHE_TTL] + --broker.address= Service address endpoint (default: http://localhost:8080) [$BROKER_ADDRESS] + --broker.cert-file= Path to the client TLS certificate [$BROKER_CERT_FILE] + --broker.cert-key-file= Path to the client TLS private key [$BROKER_CERT_KEY_FILE] + --broker.trusted-ca-file= Path to the trusted CA for client verification of server certificates + [$BROKER_TRUSTED_CA_FILE] + --broker.auth-keys= Whitespace or comma separated, base64-encoded keys. The first key is used to sign + Authorization tokens. [$BROKER_AUTH_KEYS] + --broker.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) + (default: 0) [$BROKER_CACHE_SIZE] + --broker.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$BROKER_CACHE_TTL] [edit command options] - -l, --selector= Label Selector query to filter on - --max-txn-size= maximum number of specs to be processed within an apply transaction. If 0, the default, all changes are issued in a single transaction - (default: 0) + -l, --selector= Label Selector query to filter on + --max-txn-size= maximum number of specs to be processed within an apply transaction. If 0, the default, + all changes are issued in a single transaction (default: 0) diff --git a/docs/_static/cmd-gazctl-shards-list.txt b/docs/_static/cmd-gazctl-shards-list.txt index 452e306f..fd46b5b7 100644 --- a/docs/_static/cmd-gazctl-shards-list.txt +++ b/docs/_static/cmd-gazctl-shards-list.txt @@ -19,34 +19,48 @@ It's recommended that --lag be used with a relatively focused --selector, as fetching consumption lag for a large number of shards may take a while. -Application Options: - --zone= Availability zone within which this process is running (default: local) [$ZONE] - -Logging: - --log.level=[info|debug|warn] Logging level (default: info) [$LOG_LEVEL] - --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] - Help Options: - -h, --help Show this help message + -h, --help Show this help message [shards command options] + Interact with consumer shards: + --zone= Availability zone within which this process is running (default: local) [$ZONE] + + Logging: + --log.level=[trace|debug|info|warn|error|fatal] Logging level (default: warn) [$LOG_LEVEL] + --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] + Consumer: - --consumer.address= Service address endpoint (default: http://localhost:8080) [$CONSUMER_ADDRESS] - --consumer.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) (default: 0) [$CONSUMER_CACHE_SIZE] - --consumer.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$CONSUMER_CACHE_TTL] + --consumer.address= Service address endpoint (default: http://localhost:8080) [$CONSUMER_ADDRESS] + --consumer.cert-file= Path to the client TLS certificate [$CONSUMER_CERT_FILE] + --consumer.cert-key-file= Path to the client TLS private key [$CONSUMER_CERT_KEY_FILE] + --consumer.trusted-ca-file= Path to the trusted CA for client verification of server certificates + [$CONSUMER_TRUSTED_CA_FILE] + --consumer.auth-keys= Whitespace or comma separated, base64-encoded keys. The first key is used to sign + Authorization tokens. [$CONSUMER_AUTH_KEYS] + --consumer.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) + (default: 0) [$CONSUMER_CACHE_SIZE] + --consumer.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$CONSUMER_CACHE_TTL] Broker: - --broker.address= Service address endpoint (default: http://localhost:8080) [$BROKER_ADDRESS] - --broker.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) (default: 0) [$BROKER_CACHE_SIZE] - --broker.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$BROKER_CACHE_TTL] + --broker.address= Service address endpoint (default: http://localhost:8080) [$BROKER_ADDRESS] + --broker.cert-file= Path to the client TLS certificate [$BROKER_CERT_FILE] + --broker.cert-key-file= Path to the client TLS private key [$BROKER_CERT_KEY_FILE] + --broker.trusted-ca-file= Path to the trusted CA for client verification of server certificates + [$BROKER_TRUSTED_CA_FILE] + --broker.auth-keys= Whitespace or comma separated, base64-encoded keys. The first key is used to sign + Authorization tokens. [$BROKER_AUTH_KEYS] + --broker.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) + (default: 0) [$BROKER_CACHE_SIZE] + --broker.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$BROKER_CACHE_TTL] [list command options] - -l, --selector= Label Selector query to filter on - -o, --format=[table|yaml|json|proto] Output format (default: table) - -L, --label-columns= Labels to present as columns, eg -L label-one -L label-two - -p, --primary Show primary column - -r, --replicas Show replicas column - --rf Show replication factor column - --lag Show the amount of unread data for each shard + -l, --selector= Label Selector query to filter on + -o, --format=[table|yaml|json|proto] Output format (default: table) + -L, --label-columns= Labels to present as columns, eg -L label-one -L label-two + -p, --primary Show primary column + -r, --replicas Show replicas column + --rf Show replication factor column + --lag Show the amount of unread data for each shard diff --git a/docs/_static/cmd-gazctl-shards-prune.txt b/docs/_static/cmd-gazctl-shards-prune.txt index 8a382105..3e363f34 100644 --- a/docs/_static/cmd-gazctl-shards-prune.txt +++ b/docs/_static/cmd-gazctl-shards-prune.txt @@ -10,30 +10,53 @@ Prune log examines the provided hints to identify Fragments of the log which have no intersection with any live files of the DB, and can thus be safely deleted. +CAUTION: -Application Options: - --zone= Availability zone within which this process is running (default: local) [$ZONE] +When pruning recovery logs which have been forked from other logs, +it's crucial that *all* shards which participate in the forked log +history are included in the prune operation. When pruning a log, +hints from all referencing shards are inspected to determine if a +fragment is overlapped, and a failure to include a shard which +references the log may cause data it depends on to be deleted. -Logging: - --log.level=[info|debug|warn] Logging level (default: info) [$LOG_LEVEL] - --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] Help Options: - -h, --help Show this help message + -h, --help Show this help message [shards command options] + Interact with consumer shards: + --zone= Availability zone within which this process is running (default: local) [$ZONE] + + Logging: + --log.level=[trace|debug|info|warn|error|fatal] Logging level (default: warn) [$LOG_LEVEL] + --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] + Consumer: - --consumer.address= Service address endpoint (default: http://localhost:8080) [$CONSUMER_ADDRESS] - --consumer.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) (default: 0) [$CONSUMER_CACHE_SIZE] - --consumer.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$CONSUMER_CACHE_TTL] + --consumer.address= Service address endpoint (default: http://localhost:8080) [$CONSUMER_ADDRESS] + --consumer.cert-file= Path to the client TLS certificate [$CONSUMER_CERT_FILE] + --consumer.cert-key-file= Path to the client TLS private key [$CONSUMER_CERT_KEY_FILE] + --consumer.trusted-ca-file= Path to the trusted CA for client verification of server certificates + [$CONSUMER_TRUSTED_CA_FILE] + --consumer.auth-keys= Whitespace or comma separated, base64-encoded keys. The first key is used to sign + Authorization tokens. [$CONSUMER_AUTH_KEYS] + --consumer.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) + (default: 0) [$CONSUMER_CACHE_SIZE] + --consumer.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$CONSUMER_CACHE_TTL] Broker: - --broker.address= Service address endpoint (default: http://localhost:8080) [$BROKER_ADDRESS] - --broker.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) (default: 0) [$BROKER_CACHE_SIZE] - --broker.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$BROKER_CACHE_TTL] + --broker.address= Service address endpoint (default: http://localhost:8080) [$BROKER_ADDRESS] + --broker.cert-file= Path to the client TLS certificate [$BROKER_CERT_FILE] + --broker.cert-key-file= Path to the client TLS private key [$BROKER_CERT_KEY_FILE] + --broker.trusted-ca-file= Path to the trusted CA for client verification of server certificates + [$BROKER_TRUSTED_CA_FILE] + --broker.auth-keys= Whitespace or comma separated, base64-encoded keys. The first key is used to sign + Authorization tokens. [$BROKER_AUTH_KEYS] + --broker.cache.size= Size of client route cache. If <= zero, no cache is used (server always proxies) + (default: 0) [$BROKER_CACHE_SIZE] + --broker.cache.ttl= Time-to-live of route cache entries. (default: 1m) [$BROKER_CACHE_TTL] [prune command options] - -l, --selector= Label Selector query to filter on - --dry-run Perform a dry-run of the apply + -l, --selector= Label Selector query to filter on + --dry-run Perform a dry-run of the apply diff --git a/docs/_static/cmd-gazctl.txt b/docs/_static/cmd-gazctl.txt index 1c355c2d..451b545e 100644 --- a/docs/_static/cmd-gazctl.txt +++ b/docs/_static/cmd-gazctl.txt @@ -9,15 +9,8 @@ or with '~/.config/gazette/gazctl.ini'. Use the 'print-config' sub-command to in the tool's current configuration. -Application Options: - --zone= Availability zone within which this process is running (default: local) [$ZONE] - -Logging: - --log.level=[info|debug|warn] Logging level (default: info) [$LOG_LEVEL] - --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] - Help Options: - -h, --help Show this help message + -h, --help Show this help message Available commands: attach-uuids Generate and attach UUIDs to text input records diff --git a/docs/_static/cmd-gazette-print-config.txt b/docs/_static/cmd-gazette-print-config.txt index 3f3e8d6d..13257f3b 100644 --- a/docs/_static/cmd-gazette-print-config.txt +++ b/docs/_static/cmd-gazette-print-config.txt @@ -7,25 +7,58 @@ its responsible journals and will exit only when it can safely do so. Broker: - --broker.zone= Availability zone within which this process is running (default: local) [$BROKER_ZONE] - --broker.id= Unique ID of this process. Auto-generated if not set [$BROKER_ID] - --broker.host= Addressable, advertised hostname or IP of this process. Hostname is used if not set [$BROKER_HOST] - --broker.port= Service port for HTTP and gRPC requests. A random port is used if not set [$BROKER_PORT] - --broker.limit= Maximum number of Journals the broker will allocate (default: 1024) [$BROKER_LIMIT] - --broker.file-root= Local path which roots file:// fragment stores (optional) [$BROKER_FILE_ROOT] - --broker.max-append-rate= Max rate (in bytes-per-sec) that any one journal may be appended to. If zero, there is no max rate (default: 0) [$BROKER_MAX_APPEND_RATE] - --broker.min-append-rate= Min rate (in bytes-per-sec) at which a client may stream Append RPC content. RPCs unable to sustain this rate are aborted (default: 65536) - [$BROKER_MIN_APPEND_RATE] + --broker.zone= Availability zone within which this process is running (default: local) [$BROKER_ZONE] + --broker.id= Unique ID of this process. Auto-generated if not set [$BROKER_ID] + --broker.host= Addressable, advertised hostname or IP of this process. Hostname is used if not set + [$BROKER_HOST] + --broker.port= Service port for HTTP and gRPC requests. A random port is used if not set. Port may also + take the form 'unix:///path/to/socket' to use a Unix Domain Socket [$BROKER_PORT] + --broker.server-cert-file= Path to the server TLS certificate. This option toggles whether TLS is used. If absent, all + other TLS settings are ignored. [$BROKER_SERVER_CERT_FILE] + --broker.server-cert-key-file= Path to the server TLS private key [$BROKER_SERVER_CERT_KEY_FILE] + --broker.server-ca-file= Path to the trusted CA for server verification of client certificates. When present, client + certificates are required and verified against this CA. When absent, client certificates are + not required but are verified against the system CA pool if presented. + [$BROKER_SERVER_CA_FILE] + --broker.peer-cert-file= Path to the client TLS certificate for peer-to-peer requests [$BROKER_PEER_CERT_FILE] + --broker.peer-cert-key-file= Path to the client TLS private key for peer-to-peer requests [$BROKER_PEER_CERT_KEY_FILE] + --broker.peer-ca-file= Path to the trusted CA for client verification of peer server certificates. When absent, the + system CA pool is used instead. [$BROKER_PEER_CA_FILE] + --broker.max-grpc-recv-size= Maximum size of gRPC messages accepted by this server, in bytes (default: 4194304) + [$BROKER_MAX_GRPC_RECV_SIZE] + --broker.allow-origin= Origin to allow in CORS contexts [$BROKER_ALLOW_ORIGIN] + --broker.limit= Maximum number of Journals the broker will allocate (default: 1024) [$BROKER_LIMIT] + --broker.file-root= Local path which roots file:// fragment stores (optional) [$BROKER_FILE_ROOT] + --broker.file-only Use the local file:// store for all journal fragments, ignoring cloud bucket storage + configuration (for example, S3) [$BROKER_FILE_ONLY] + --broker.max-append-rate= Max rate (in bytes-per-sec) that any one journal may be appended to. If zero, there is no + max rate (default: 0) [$BROKER_MAX_APPEND_RATE] + --broker.max-replication= Maximum effective replication of any one journal, which upper-bounds its stated replication. + (default: 9) [$BROKER_MAX_REPLICATION] + --broker.min-append-rate= Min rate (in bytes-per-sec) at which a client may stream Append RPC content. RPCs unable to + sustain this rate are aborted (default: 65536) [$BROKER_MIN_APPEND_RATE] + --broker.watch-delay= Delay applied to the application of watched Etcd events. Larger values amortize the + processing of fast-changing Etcd keys. (default: 30ms) [$BROKER_WATCH_DELAY] + --broker.auth-keys= Whitespace or comma separated, base64-encoded keys used to sign (first key) and verify (all + keys) Authorization tokens. [$BROKER_AUTH_KEYS] + --broker.auto-suspend Automatically suspend journals which have persisted all fragments [$BROKER_AUTO_SUSPEND] Etcd: - --etcd.address= Etcd service address endpoint (default: http://localhost:2379) [$ETCD_ADDRESS] - --etcd.lease= Time-to-live of Etcd lease (default: 20s) [$ETCD_LEASE_TTL] - --etcd.prefix= Etcd base prefix for broker state and coordination (default: /gazette/cluster) [$ETCD_PREFIX] + --etcd.address= Etcd service address endpoint (default: http://localhost:2379) [$ETCD_ADDRESS] + --etcd.cert-file= Path to the client TLS certificate [$ETCD_CERT_FILE] + --etcd.cert-key-file= Path to the client TLS private key [$ETCD_CERT_KEY_FILE] + --etcd.trusted-ca-file= Path to the trusted CA for client verification of server certificates [$ETCD_TRUSTED_CA_FILE] + --etcd.lease= Time-to-live of Etcd lease (default: 20s) [$ETCD_LEASE_TTL] + --etcd.prefix= Etcd base prefix for broker state and coordination (default: /gazette/cluster) [$ETCD_PREFIX] Logging: - --log.level=[info|debug|warn] Logging level (default: info) [$LOG_LEVEL] - --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] + --log.level=[trace|debug|info|warn|error|fatal] Logging level (default: warn) [$LOG_LEVEL] + --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] + +Debug: + --debug.port= Port for diagnostics [$DEBUG_PORT] + --debug.private When true, don't serve diagnostics over the main service port [$DEBUG_PRIVATE] Help Options: - -h, --help Show this help message + -h, --help Show this help message diff --git a/docs/_static/cmd-gazette-serve.txt b/docs/_static/cmd-gazette-serve.txt index 3f3e8d6d..13257f3b 100644 --- a/docs/_static/cmd-gazette-serve.txt +++ b/docs/_static/cmd-gazette-serve.txt @@ -7,25 +7,58 @@ its responsible journals and will exit only when it can safely do so. Broker: - --broker.zone= Availability zone within which this process is running (default: local) [$BROKER_ZONE] - --broker.id= Unique ID of this process. Auto-generated if not set [$BROKER_ID] - --broker.host= Addressable, advertised hostname or IP of this process. Hostname is used if not set [$BROKER_HOST] - --broker.port= Service port for HTTP and gRPC requests. A random port is used if not set [$BROKER_PORT] - --broker.limit= Maximum number of Journals the broker will allocate (default: 1024) [$BROKER_LIMIT] - --broker.file-root= Local path which roots file:// fragment stores (optional) [$BROKER_FILE_ROOT] - --broker.max-append-rate= Max rate (in bytes-per-sec) that any one journal may be appended to. If zero, there is no max rate (default: 0) [$BROKER_MAX_APPEND_RATE] - --broker.min-append-rate= Min rate (in bytes-per-sec) at which a client may stream Append RPC content. RPCs unable to sustain this rate are aborted (default: 65536) - [$BROKER_MIN_APPEND_RATE] + --broker.zone= Availability zone within which this process is running (default: local) [$BROKER_ZONE] + --broker.id= Unique ID of this process. Auto-generated if not set [$BROKER_ID] + --broker.host= Addressable, advertised hostname or IP of this process. Hostname is used if not set + [$BROKER_HOST] + --broker.port= Service port for HTTP and gRPC requests. A random port is used if not set. Port may also + take the form 'unix:///path/to/socket' to use a Unix Domain Socket [$BROKER_PORT] + --broker.server-cert-file= Path to the server TLS certificate. This option toggles whether TLS is used. If absent, all + other TLS settings are ignored. [$BROKER_SERVER_CERT_FILE] + --broker.server-cert-key-file= Path to the server TLS private key [$BROKER_SERVER_CERT_KEY_FILE] + --broker.server-ca-file= Path to the trusted CA for server verification of client certificates. When present, client + certificates are required and verified against this CA. When absent, client certificates are + not required but are verified against the system CA pool if presented. + [$BROKER_SERVER_CA_FILE] + --broker.peer-cert-file= Path to the client TLS certificate for peer-to-peer requests [$BROKER_PEER_CERT_FILE] + --broker.peer-cert-key-file= Path to the client TLS private key for peer-to-peer requests [$BROKER_PEER_CERT_KEY_FILE] + --broker.peer-ca-file= Path to the trusted CA for client verification of peer server certificates. When absent, the + system CA pool is used instead. [$BROKER_PEER_CA_FILE] + --broker.max-grpc-recv-size= Maximum size of gRPC messages accepted by this server, in bytes (default: 4194304) + [$BROKER_MAX_GRPC_RECV_SIZE] + --broker.allow-origin= Origin to allow in CORS contexts [$BROKER_ALLOW_ORIGIN] + --broker.limit= Maximum number of Journals the broker will allocate (default: 1024) [$BROKER_LIMIT] + --broker.file-root= Local path which roots file:// fragment stores (optional) [$BROKER_FILE_ROOT] + --broker.file-only Use the local file:// store for all journal fragments, ignoring cloud bucket storage + configuration (for example, S3) [$BROKER_FILE_ONLY] + --broker.max-append-rate= Max rate (in bytes-per-sec) that any one journal may be appended to. If zero, there is no + max rate (default: 0) [$BROKER_MAX_APPEND_RATE] + --broker.max-replication= Maximum effective replication of any one journal, which upper-bounds its stated replication. + (default: 9) [$BROKER_MAX_REPLICATION] + --broker.min-append-rate= Min rate (in bytes-per-sec) at which a client may stream Append RPC content. RPCs unable to + sustain this rate are aborted (default: 65536) [$BROKER_MIN_APPEND_RATE] + --broker.watch-delay= Delay applied to the application of watched Etcd events. Larger values amortize the + processing of fast-changing Etcd keys. (default: 30ms) [$BROKER_WATCH_DELAY] + --broker.auth-keys= Whitespace or comma separated, base64-encoded keys used to sign (first key) and verify (all + keys) Authorization tokens. [$BROKER_AUTH_KEYS] + --broker.auto-suspend Automatically suspend journals which have persisted all fragments [$BROKER_AUTO_SUSPEND] Etcd: - --etcd.address= Etcd service address endpoint (default: http://localhost:2379) [$ETCD_ADDRESS] - --etcd.lease= Time-to-live of Etcd lease (default: 20s) [$ETCD_LEASE_TTL] - --etcd.prefix= Etcd base prefix for broker state and coordination (default: /gazette/cluster) [$ETCD_PREFIX] + --etcd.address= Etcd service address endpoint (default: http://localhost:2379) [$ETCD_ADDRESS] + --etcd.cert-file= Path to the client TLS certificate [$ETCD_CERT_FILE] + --etcd.cert-key-file= Path to the client TLS private key [$ETCD_CERT_KEY_FILE] + --etcd.trusted-ca-file= Path to the trusted CA for client verification of server certificates [$ETCD_TRUSTED_CA_FILE] + --etcd.lease= Time-to-live of Etcd lease (default: 20s) [$ETCD_LEASE_TTL] + --etcd.prefix= Etcd base prefix for broker state and coordination (default: /gazette/cluster) [$ETCD_PREFIX] Logging: - --log.level=[info|debug|warn] Logging level (default: info) [$LOG_LEVEL] - --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] + --log.level=[trace|debug|info|warn|error|fatal] Logging level (default: warn) [$LOG_LEVEL] + --log.format=[json|text|color] Logging output format (default: text) [$LOG_FORMAT] + +Debug: + --debug.port= Port for diagnostics [$DEBUG_PORT] + --debug.private When true, don't serve diagnostics over the main service port [$DEBUG_PRIVATE] Help Options: - -h, --help Show this help message + -h, --help Show this help message diff --git a/docs/overview-build-and-test.rst b/docs/overview-build-and-test.rst index da66a0c5..107745f1 100644 --- a/docs/overview-build-and-test.rst +++ b/docs/overview-build-and-test.rst @@ -5,8 +5,6 @@ Most binaries and packages are "pure" Go and can be directly go-installed and go .. code-block:: console - $ export GO111MODULE=on - $ go install go.gazette.dev/core/cmd/gazette $ go install go.gazette.dev/core/cmd/gazctl $ go test go.gazette.dev/core/broker/... @@ -14,8 +12,8 @@ Most binaries and packages are "pure" Go and can be directly go-installed and go Certain packages used by consumer applications, like ``go.gazette.dev/core/consumer/store-rocksdb``, require CGO to build and also require appropriate development libraries for RocksDB. -Standard Linux packages are insufficient, as run-time type information must be enabled -in the RocksDB build (and it's turned off in the standard Debian package, for example). +On Debian and Ubuntu, the ``librocksdb-dev`` and ``libsqlite3-dev`` packages are sufficient, +paired with the ``libsqlite3`` Go build tag. Continuous Integration @@ -24,93 +22,19 @@ Continuous Integration Gazette uses a :githubsource:`Make-based build system` which pulls down and stages development dependencies into a ``.build`` sub-directory of the repository root. -The Make build system offers fully hermetic builds using a Docker-in-Docker -builder. Run CI tests in a hermetic build environment: +Run CI tests as: .. code-block:: console - $ make as-ci target=go-test-ci + $ make go-test-ci Continuous integration builds of Gazette run tests 15 times, with race detection enabled. -Package release Docker images for the Gazette broker and examples, -as ``gazette/broker:latest`` and ``gazette/examples:latest``. - -.. code-block:: console - - $ make as-ci target=ci-release-gazette-broker - $ make as-ci target=ci-release-gazette-examples - -Deploy Gazette's continuous soak test to a Kubernetes cluster (which can be -Docker for Desktop or Minikube). Soak tests run with ``latest`` images. - -.. code-block:: console - - $ kubectl apply -k ./kustomize/test/deploy-stream-sum-with-crash-tests/ - -Push images to a registry. If ``REGISTRY`` is not specified, it defaults to ``localhost:32000`` (which is used by MicroK8s): - -.. code-block:: console - - $ make push-to-registry REGISTRY=my.registry.com - -The ``kustomize`` directory also has a -:githubsource:`helper manifest` for using -a local registry (eg, for development builds). - - -Dependencies -------------- - -It's recommended to use the docker-based build targets described above in most situations, since the -docker image will have all the required dependencies. If you want to execute build targets directly -instead of using ``as-ci``, then the following dependencies are required: +A ``go-test-fast`` target is also available, wich runs tests once without race detection. -* Compression library development headers ("-dev" packages): - - * ``libbz2`` - * ``libsnappy`` - * ``liblz4`` - * ``libzstd`` - * ``libbz2`` - -* Protobuf compiler: - - * ``libprotobuf`` (development headers) - * ``protobuf-compiler`` - -* Etcd: The deb/rpm packages provided in many repositories are likely too old to work with - Gazette. Gazette requires version 3.3 or later. Version 3.4.x is recommended, since that is used - in Gazette CI. - -* Sqlite - - * ``libsqlite3`` (development headers) - * It's also probably useful to have the sqlite3 CLI for debugging - -* RocksDB: On linux systems, this will be downloaded and built automatically. You'll need to have a - few things in order for this to work. Most systems will already have this stuff, but it's listed - here anyway just for the sake of being thorough - - * A C compiler toolchain (on debian-based distros, the ``build-essential`` package will have you covered) - * ``curl`` - * ``ca-certificates`` (so that curl can validate the certificate of the rocksdb download server) - * ``tar`` - -Other Build Targets --------------------- - -If you execute these targets directly, then you'll need to have all of the above dependencies installed. - -.. code-block:: console - - $ make go-install - $ make go-test-fast - - -.. code-block:: console +Build release binaries for Gazette broker and examples using +``go-build`` or ``go-build-arm64`` targets. - $ make go-test-ci Building the Docs ------------------ diff --git a/docs/reference-gazctl.rst b/docs/reference-gazctl.rst index e4125f18..56a07222 100644 --- a/docs/reference-gazctl.rst +++ b/docs/reference-gazctl.rst @@ -41,6 +41,10 @@ gazctl journals reset-head --------------------------- .. literalinclude:: _static/cmd-gazctl-journals-reset-head.txt +gazctl journals suspend +--------------------------- +.. literalinclude:: _static/cmd-gazctl-journals-suspend.txt + gazctl print-config --------------------------- .. literalinclude:: _static/cmd-gazctl-print-config.txt