Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kv: prototype buffered writes #122862

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
+2,813 −17
Draft

kv: prototype buffered writes #122862

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
49298d9
sql: more blind writes
nvanbenschoten Apr 22, 2024
3288d34
kv: gen buffered writes btree
nvanbenschoten Apr 22, 2024
169a4b5
kv: prototype buffered writes
nvanbenschoten Apr 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions build/bazelutil/check.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ pkg/roachprod/prometheus/prometheus.go://go:generate mockgen -package=prometheus
pkg/cmd/roachtest/clusterstats/collector.go://go:generate mockgen -package=clusterstats -destination mocks_generated_test.go github.com/cockroachdb/cockroach/pkg/roachprod/prometheus Client
pkg/cmd/roachtest/tests/drt.go://go:generate mockgen -package tests -destination drt_generated_test.go github.com/cockroachdb/cockroach/pkg/roachprod/prometheus Client
pkg/kv/kvclient/kvcoord/transport.go://go:generate mockgen -package=kvcoord -destination=mocks_generated_test.go . Transport
pkg/kv/kvclient/kvcoord/txn_interceptor_write_buffer.go://go:generate ../../../util/interval/generic/gen.sh *bufferedWrite kvcoord
pkg/kv/kvclient/rangecache/range_cache.go://go:generate mockgen -package=rangecachemock -destination=rangecachemock/mocks_generated.go . RangeDescriptorDB
pkg/kv/kvclient/rangefeed/rangefeed.go://go:generate mockgen -destination=mocks_generated_test.go --package=rangefeed . DB
pkg/kv/kvserver/concurrency/lock_table.go://go:generate ../../../util/interval/generic/gen.sh *keyLocks concurrency
Expand Down
1 change: 1 addition & 0 deletions docs/generated/settings/settings-for-tenants.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,7 @@ kv.protectedts.reconciliation.interval duration 5m0s the frequency for reconcili
kv.rangefeed.client.stream_startup_rate integer 100 controls the rate per second the client will initiate new rangefeed stream for a single range; 0 implies unlimited application
kv.rangefeed.closed_timestamp_refresh_interval duration 3s the interval at which closed-timestamp updatesare delivered to rangefeeds; set to 0 to use kv.closed_timestamp.side_transport_interval system-visible
kv.rangefeed.enabled boolean false if set, rangefeed registration is enabled system-visible
kv.transaction.buffered_writes.enabled boolean true if enabled, transactional writes are buffered on the gateway application
kv.transaction.max_intents_and_locks integer 0 maximum count of inserts or durable locks for a single transactions, 0 to disable application
kv.transaction.max_intents_bytes integer 4194304 maximum number of bytes used to track locks in transactions application
kv.transaction.max_refresh_spans_bytes integer 4194304 maximum number of bytes used to track refresh spans in serializable transactions application
Expand Down
1 change: 1 addition & 0 deletions docs/generated/settings/settings.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,7 @@
<tr><td><div id="setting-kv-replication-reports-interval" class="anchored"><code>kv.replication_reports.interval</code></div></td><td>duration</td><td><code>1m0s</code></td><td>the frequency for generating the replication_constraint_stats, replication_stats_report and replication_critical_localities reports (set to 0 to disable)</td><td>Dedicated/Self-Hosted</td></tr>
<tr><td><div id="setting-kv-snapshot-rebalance-max-rate" class="anchored"><code>kv.snapshot_rebalance.max_rate</code></div></td><td>byte size</td><td><code>32 MiB</code></td><td>the rate limit (bytes/sec) to use for rebalance and upreplication snapshots</td><td>Dedicated/Self-Hosted</td></tr>
<tr><td><div id="setting-kv-snapshot-receiver-excise-enabled" class="anchored"><code>kv.snapshot_receiver.excise.enabled</code></div></td><td>boolean</td><td><code>true</code></td><td>set to false to disable excises in place of range deletions for KV snapshots</td><td>Dedicated/Self-Hosted</td></tr>
<tr><td><div id="setting-kv-transaction-buffered-writes-enabled" class="anchored"><code>kv.transaction.buffered_writes.enabled</code></div></td><td>boolean</td><td><code>true</code></td><td>if enabled, transactional writes are buffered on the gateway</td><td>Serverless/Dedicated/Self-Hosted</td></tr>
<tr><td><div id="setting-kv-transaction-max-intents-and-locks" class="anchored"><code>kv.transaction.max_intents_and_locks</code></div></td><td>integer</td><td><code>0</code></td><td>maximum count of inserts or durable locks for a single transactions, 0 to disable</td><td>Serverless/Dedicated/Self-Hosted</td></tr>
<tr><td><div id="setting-kv-transaction-max-intents-bytes" class="anchored"><code>kv.transaction.max_intents_bytes</code></div></td><td>integer</td><td><code>4194304</code></td><td>maximum number of bytes used to track locks in transactions</td><td>Serverless/Dedicated/Self-Hosted</td></tr>
<tr><td><div id="setting-kv-transaction-max-refresh-spans-bytes" class="anchored"><code>kv.transaction.max_refresh_spans_bytes</code></div></td><td>integer</td><td><code>4194304</code></td><td>maximum number of bytes used to track refresh spans in serializable transactions</td><td>Serverless/Dedicated/Self-Hosted</td></tr>
Expand Down
2 changes: 2 additions & 0 deletions pkg/gen/misc.bzl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ MISC_SRCS = [
"//pkg/backup:data_driven_generated_test.go",
"//pkg/ccl/kvccl/kvtenantccl/upgradeinterlockccl:generated_test.go",
"//pkg/internal/team:TEAMS.yaml",
"//pkg/kv/kvclient/kvcoord:bufferedwrite_interval_btree.go",
"//pkg/kv/kvclient/kvcoord:bufferedwrite_interval_btree_test.go",
"//pkg/kv/kvpb:batch_generated.go",
"//pkg/kv/kvserver/concurrency:keylocks_interval_btree.go",
"//pkg/kv/kvserver/concurrency:keylocks_interval_btree_test.go",
Expand Down
10 changes: 10 additions & 0 deletions pkg/kv/kvclient/kvcoord/BUILD.bazel
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ load("@bazel_gomock//:gomock.bzl", "gomock")
load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test")
load("//build:STRINGER.bzl", "stringer")
load("//pkg/testutils:buildutil/buildutil.bzl", "disallowed_imports_test")
load("//pkg/util/interval/generic:gen.bzl", "gen_interval_btree")

go_library(
name = "kvcoord",
Expand Down Expand Up @@ -31,8 +32,10 @@ go_library(
"txn_interceptor_pipeliner.go",
"txn_interceptor_seq_num_allocator.go",
"txn_interceptor_span_refresher.go",
"txn_interceptor_write_buffer.go",
"txn_lock_gatekeeper.go",
"txn_metrics.go",
":bufferedwrite_interval_btree.go", # keep
":gen-txnstate-stringer", # keep
],
importpath = "github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord",
Expand Down Expand Up @@ -152,6 +155,7 @@ go_test(
"txn_interceptor_seq_num_allocator_test.go",
"txn_interceptor_span_refresher_test.go",
"txn_test.go",
":bufferedwrite_interval_btree.go", # keep
":mock_kvcoord", # keep
],
data = glob(["testdata/**"]),
Expand Down Expand Up @@ -255,6 +259,12 @@ stringer(
typ = "txnState",
)

gen_interval_btree(
name = "buffered_write_interval_btree",
package = "kvcoord",
type = "*bufferedWrite",
)

disallowed_imports_test(
"kvcoord",
disallowed_list = [
Expand Down
Loading
Loading