Skip to content
This repository has been archived by the owner on Apr 2, 2024. It is now read-only.

Memory map kafka checkpoint file #1950

Open
wants to merge 1 commit into
base: dev
Choose a base branch
from

Conversation

andremedeiros
Copy link

Problem

Heka's KafkaInput plugin writes the offset for every Kafka message. When it's a low volume topic, this doesn't cause any issues, but with high volume ones, disk IO becomes a limiting factor to how quickly Heka can process messages.

Solution

Instead of writing the offset on every message to disk, this PR mmaps it. A quick benchmark shows the advantages of using this method:

BenchmarkFile-8                  10000         119687 ns/op
BenchmarkMmap-8             2000000000           0.64 ns/op

No new tests were written, and the ones that verify checkpoint correctness still pass.

r. @fbogsany @rafrombrc

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant