Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify the bytes decoding of chunks #607

Merged
merged 4 commits into from
Jun 18, 2024

Conversation

jianoaix
Copy link
Contributor

@jianoaix jianoaix commented Jun 18, 2024

Why are these changes needed?

Simplicity is better than complicatedness

This PR also makes it 15x faster to decode the chunks (BenchmarkDecocodeChunksOld is before this PR, BenchmarkDecocodeChunks is after this PR)

goos: linux
goarch: amd64
pkg: github.com/Layr-Labs/eigenda/node
cpu: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
BenchmarkEncodeChunks-8        	  198956	      5647 ns/op
BenchmarkEncodeChunks-8        	  206421	      5404 ns/op
BenchmarkEncodeChunks-8        	  223056	      5520 ns/op
BenchmarkEncodeChunks-8        	  203401	      5757 ns/op
BenchmarkDecocodeChunksOld-8   	  122836	      9562 ns/op
BenchmarkDecocodeChunksOld-8   	  127582	      9511 ns/op
BenchmarkDecocodeChunksOld-8   	  126115	      9490 ns/op
BenchmarkDecocodeChunksOld-8   	  126057	      9575 ns/op
BenchmarkDecocodeChunks-8      	 1974146	       604.8 ns/op
BenchmarkDecocodeChunks-8      	 1996282	       604.3 ns/op
BenchmarkDecocodeChunks-8      	 1978239	       606.9 ns/op
BenchmarkDecocodeChunks-8      	 1981912	       604.5 ns/op
PASS
ok  	github.com/Layr-Labs/eigenda/node	17.480s

Checks

  • I've made sure the lint is passing in this PR.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, in that case, please comment that they are not relevant.
  • Testing Strategy
    • Unit tests
    • Integration tests
    • This PR is not tested :(

@jianoaix jianoaix requested review from dmanc, bxue-l2 and ian-shim June 18, 2024 22:05
}
chunk := buf[:chunkSize]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should actually also be much more efficient, as this doesn't need to copy chunks (no memory allocation).

for i := 0; i < b.N; i++ {
_, _ = node.DecodeChunks(sampleChunks[i%numSamples])
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed that there isn't any test that validates chunks encoded by EncodeChunks are recovered by DecodeChunks. Can we add it to make sure encoding/decoding is consistent?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was indirectly tested by other tests which need to get chunks (e.g. TestRetrieveChunks). But yep it's good to cover this with more rigorous testing.

@jianoaix jianoaix merged commit 7c8751a into Layr-Labs:master Jun 18, 2024
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants