Skip to content

Commit

Permalink
producer_worker: revert use of ValueGenerator::Generate()
Browse files Browse the repository at this point in the history
Using the modified `Generate()` function was a lot slower than just simply making
an empty payload:

```
Benchmark_random_payload10-32    2445903       485.2 ns/op
Benchmark_random_payload100-32    678870        1773 ns/op
Benchmark_random_payload1000-32   105667       13700 ns/op
Benchmark_random_payload1e4-32      9104      126782 ns/op
Benchmark_random_payload1e5-32      1135     1151425 ns/op
Benchmark_random_payload1e6-32       100    13046547 ns/op
==========================================================
Benchmark_empty_payload10-32    79950942       20.54 ns/op
Benchmark_empty_payload100-32   24089101       54.78 ns/op
Benchmark_empty_payload1000-32   3789248       366.1 ns/op
Benchmark_empty_payload1e4-32     670096        2059 ns/op
Benchmark_empty_payload1e5-32      58912       23021 ns/op
Benchmark_empty_payload1e6-32       5600      290045 ns/op
```

As can be seen, generating the random payload and ensuring it is UTF-8 valid
is orders of magnitude slower than the empty payload.

Revert the use of `Generate()` here in place of the empty payload.

However, if the user has indicated they want to validate the latest key-value
pair produced, generate a `(value-{%018d}, offset)` message in the record.

This does mean that message sizes that are less than 24 bytes are not honored
if the `validate-latest-values` flag is passed.
  • Loading branch information
WillemKauf committed Dec 12, 2024
1 parent bffac1f commit 0dfae76
Show file tree
Hide file tree
Showing 2 changed files with 17 additions and 6 deletions.
19 changes: 17 additions & 2 deletions pkg/worker/verifier/producer_worker.go
Original file line number Diff line number Diff line change
Expand Up @@ -121,10 +121,25 @@ func (pw *ProducerWorker) newRecord(producerId int, sequence int64) *kgo.Record
pw.Status.AbortedTransactionMessages += 1
}

payload := pw.config.valueGenerator.Generate()
if payload == nil {
var payload []byte
isTombstone := rand.Float64() < pw.config.valueGenerator.TombstoneProbability
if isTombstone {
payload = nil
pw.Status.TombstonesProduced += 1
} else {
if pw.validateLatestValues {
var value bytes.Buffer
fmt.Fprintf(&value, "value-%018d", sequence)
paddingSize := pw.config.messageSize - value.Len()
if paddingSize > 0 {
value.Write(make([]byte, paddingSize))
}
payload = value.Bytes()
} else {
payload = make([]byte, pw.config.messageSize)
}
}

var r *kgo.Record

if pw.config.keySetCardinality < 0 {
Expand Down
4 changes: 0 additions & 4 deletions pkg/worker/worker.go
Original file line number Diff line number Diff line change
Expand Up @@ -108,10 +108,6 @@ type ValueGenerator struct {
var compressible_payload []byte

func (vg *ValueGenerator) Generate() []byte {
isTombstone := rand.Float64() < vg.TombstoneProbability
if isTombstone {
return nil
}
if vg.Compressible {
// Zeros, which is about as compressible as an array can be.
if len(compressible_payload) == 0 {
Expand Down

0 comments on commit 0dfae76

Please sign in to comment.