-
Notifications
You must be signed in to change notification settings - Fork 193
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement core relay functionality #869
Conversation
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
relay/chunk_manager.go
Outdated
fMap := make(frameMap, len(keys)) | ||
hadError := atomic.Bool{} | ||
|
||
wg := sync.WaitGroup{} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same comment about the wg and errgroup
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Constructor changed. Let me know if you'd like to move away from the errgroup abstraction, and I'll change both locations.
Signed-off-by: Cody Littley <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
initial review, will give it another go tomorrow
relay/blob_manager.go
Outdated
func (s *blobManager) GetBlob(blobKey v2.BlobKey) ([]byte, error) { | ||
|
||
// Even though we don't need extra parallelism here, we still use the work pool to ensure that we don't | ||
// permit too many concurrent requests to the blob store. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What happens if there are too many concurrent requests to S3? I'm wondering if it could be worth using this blob manager on the encoder (although it doesn't need the caching part).
See:
eigenda/disperser/encoder/server_v2.go
Line 128 in 94ab42f
data, err := s.blobStore.GetBlob(ctx, blobKey) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nothing catastrophic if there is too much S3 traffic, other than a potential slowdown. After a certain point, adding more goroutines attempting to pull data will result in a net reduction in capacity. Think of this sort of code as a defensive programming in case the node is overloaded with work.
relay/config.go
Outdated
// serve data for any shard it can. | ||
Shards []core.RelayKey | ||
|
||
// MetadataCacheSize is the size of the metadata cache. Default is 1024 * 1024. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we add some units to the config descriptions?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Description added.
// MetadataCacheSize is the size of the metadata cache. Current cache implementation is unaware of data sizes, and
// so this is a total count, not a size in bytes. Default is 1024 * 1024.
MetadataCacheSize int
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
@ian-shim Check again. Forgot to push. 🙃 |
Signed-off-by: Cody Littley <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Few more comments. LGTM otherwise
relay/chunk_manager.go
Outdated
type frameMap map[v2.BlobKey][]*encoding.Frame | ||
|
||
// GetFrames retrieves the frames for a blob. | ||
func (s *chunkManager) GetFrames(ctx context.Context, mMap *metadataMap) (*frameMap, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maps are already reference types, so don't need to pass a pointer
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All map pointers converted to non-pointer type
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm, please address the remaining comments
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
left some minor comments but lgtm
relay/blob_provider.go
Outdated
if err != nil { | ||
// It should not be possible for external users to force an error here since we won't | ||
// even call this method if the blob key is invalid (so it's ok to have a noisy log here). | ||
s.logger.Error("Failed to fetch blob: %v", err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should it be errorf?, same comment for fetchBlob
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch, fixed
relay/chunk_provider.go
Outdated
go func() { | ||
frames, err := s.frameCache.Get(*boundKey) | ||
if err != nil { | ||
s.logger.Error("Failed to get frames for blob %v: %v", boundKey.blobKey, err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Why are these changes needed?
This PR adds core functionality for the relay server. The following features have intentionally been omitted, and will be merged as follow-up PRs:
Checks