Merge branch 'develop' into remote-signer-electra

CHANGELOG.md

@@ -4,7 +4,79 @@ All notable changes to this project will be documented in this file.
 
 The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
 
-## [Unreleased](https://github.com/prysmaticlabs/prysm/compare/v5.1.0...HEAD)
+## [Unreleased](https://github.com/prysmaticlabs/prysm/compare/v5.1.2...HEAD)
+
+### Added
+
+- Electra EIP6110: Queue deposit [pr](https://github.com/prysmaticlabs/prysm/pull/14430)
+- Add Bellatrix tests for light client functions.
+- Add Discovery Rebooter Feature.
+- Added GetBlockAttestationsV2 endpoint.
+- Light client support: Consensus types for Electra
+- Added SubmitPoolAttesterSlashingV2 endpoint.
+- Added SubmitAggregateAndProofsRequestV2 endpoint.
+- Updated the `beacon-chain/monitor` package to Electra. [PR](https://github.com/prysmaticlabs/prysm/pull/14562)
+- Remote signer electra fork support.
+
+### Changed
+
+- Electra EIP6110: Queue deposit requests changes from consensus spec pr #3818
+- reversed the boolean return on `BatchVerifyDepositsSignatures`, from need verification, to all keys successfully verified
+- Fix `engine_exchangeCapabilities` implementation.
+- Updated the default `scrape-interval` in `Client-stats` to 2 minutes to accommodate Beaconcha.in API rate limits.
+- Switch to compounding when consolidating with source==target.
+- Revert block db save when saving state fails.
+- Return false from HasBlock if the block is being synced. 
+- Cleanup forkchoice on failed insertions.
+- Use read only validator for core processing to avoid unnecessary copying.
+- Use ROBlock across block processing pipeline
+
+### Deprecated
+
+- `/eth/v1alpha1/validator/activation/stream` grpc wait for activation stream is deprecated. [pr](https://github.com/prysmaticlabs/prysm/pull/14514)
+
+### Removed
+
+- Removed finalized validator index cache, no longer needed.
+
+### Fixed
+
+- Fixed mesh size by appending `gParams.Dhi = gossipSubDhi`
+- Fix skipping partial withdrawals count.
+- wait for the async StreamEvent writer to exit before leaving the http handler, avoiding race condition panics [pr](https://github.com/prysmaticlabs/prysm/pull/14557)
+
+### Security
+
+## [v5.1.2](https://github.com/prysmaticlabs/prysm/compare/v5.1.1...v5.1.2) - 2024-10-16 
+
+This is a hotfix release with one change. 
+
+Prysm v5.1.1 contains an updated implementation of the beacon api streaming events endpoint. This
+new implementation contains a bug that can cause a panic in certain conditions. The issue is
+difficult to reproduce reliably and we are still trying to determine the root cause, but in the
+meantime we are issuing a patch that recovers from the panic to prevent the node from crashing.
+
+This only impacts the v5.1.1 release beacon api event stream endpoints. This endpoint is used by the
+prysm REST mode validator (a feature which requires the validator to be configured to use the beacon
+api intead of prysm's stock grpc endpoints) or accessory software that connects to the events api,
+like https://github.com/ethpandaops/ethereum-metrics-exporter
+
+### Fixed 
+
+- Recover from panics when writing the event stream [#14545](https://github.com/prysmaticlabs/prysm/pull/14545)
+
+## [v5.1.1](https://github.com/prysmaticlabs/prysm/compare/v5.1.0...v5.1.1) - 2024-10-15
+
+This release has a number of features and improvements. Most notably, the feature flag 
+`--enable-experimental-state` has been flipped to "opt out" via `--disable-experimental-state`. 
+The experimental state management design has shown significant improvements in memory usage at
+runtime. Updates to libp2p's gossipsub have some bandwidith stability improvements with support for
+IDONTWANT control messages. 
+
+The gRPC gateway has been deprecated from Prysm in this release. If you need JSON data, consider the
+standardized beacon-APIs. 
+
+Updating to this release is recommended at your convenience.
 
 ### Added
 
@@ -13,14 +85,22 @@ The format is based on Keep a Changelog, and this project adheres to Semantic Ve
 - Light client support: Implement `ComputeFieldRootsForBlockBody`.
 - Light client support: Add light client database changes.
 - Light client support: Implement capella and deneb changes.
-- Light client support: Implement `BlockToLightClientHeaderXXX` functions upto Deneb
+- Light client support: Implement `BlockToLightClientHeader` function.
+- Light client support: Consensus types.
 - GetBeaconStateV2: add Electra case.
-- Implement [consensus-specs/3875](https://github.com/ethereum/consensus-specs/pull/3875)
-- Tests to ensure sepolia config matches the official upstream yaml
-- remote signer electra fork support.
+- Implement [consensus-specs/3875](https://github.com/ethereum/consensus-specs/pull/3875).
+- Tests to ensure sepolia config matches the official upstream yaml.
+- `engine_newPayloadV4`,`engine_getPayloadV4` used for electra payload communication with execution client.  [pr](https://github.com/prysmaticlabs/prysm/pull/14492)
+- HTTP endpoint for PublishBlobs.
+- GetBlockV2, GetBlindedBlock, ProduceBlockV2, ProduceBlockV3: add Electra case.
+- Add Electra support and tests for light client functions.
+- fastssz version bump (better error messages).
+- SSE implementation that sheds stuck clients. [pr](https://github.com/prysmaticlabs/prysm/pull/14413)
+- Added GetPoolAttesterSlashingsV2 endpoint.
 
 ### Changed
 
+- Electra: Updated interop genesis generator to support Electra.
 - `getLocalPayload` has been refactored to enable work in ePBS branch.
 - `TestNodeServer_GetPeer` and `TestNodeServer_ListPeers` test flakes resolved by iterating the whole peer list to find
   a match rather than taking the first peer in the map.
@@ -41,18 +121,23 @@ The format is based on Keep a Changelog, and this project adheres to Semantic Ve
 - Updated k8s-io/client-go to v0.30.4 and k8s-io/apimachinery to v0.30.4
 - Migrated tracing library from opencensus to opentelemetry for both the beacon node and validator.
 - Refactored light client code to make it more readable and make future PRs easier.
+- Update light client helper functions to reference `dev` branch of CL specs
 - Updated Libp2p Dependencies to allow prysm to use gossipsub v1.2 .
 - Updated Sepolia bootnodes.
+- Make committee aware packing the default by deprecating `--enable-committee-aware-packing`.
+- Moved `ConvertKzgCommitmentToVersionedHash` to the `primitives` package.
+- Updated correlation penalty for EIP-7251. 
 
 ### Deprecated
 - `--disable-grpc-gateway` flag is deprecated due to grpc gateway removal.
 - `--enable-experimental-state` flag is deprecated. This feature is now on by default. Opt-out with `--disable-experimental-state`.
 
 ### Removed
 
-- removed gRPC Gateway
-- Removed unused blobs bundle cache
+- Removed gRPC Gateway.
+- Removed unused blobs bundle cache.
 - Removed consolidation signing domain from params. The Electra design changed such that EL handles consolidation signature verification.
+- Remove engine_getPayloadBodiesBy{Hash|Range}V2
 
 ### Fixed
 
@@ -67,10 +152,15 @@ The format is based on Keep a Changelog, and this project adheres to Semantic Ve
 - Core: Fixed slash processing causing extra hashing.
 - Core: Fixed extra allocations when processing slashings.
 - remove unneeded container in blob sidecar ssz response
+- Light client support: create finalized header based on finalizedBlock's version, not attestedBlock.
+- Light client support: fix light client attested header execution fields' wrong version bug.
 - Testing: added custom matcher for better push settings testing.
+- Registered `GetDepositSnapshot` Beacon API endpoint.
 
 ### Security
 
+No notable security updates.
+
 ## [v5.1.0](https://github.com/prysmaticlabs/prysm/compare/v5.0.4...v5.1.0) - 2024-08-20
 
 This release contains 171 new changes and many of these are related to Electra! Along side the Electra changes, there

WORKSPACE

@@ -227,7 +227,7 @@ filegroup(
     url = "https://github.com/ethereum/EIPs/archive/5480440fe51742ed23342b68cf106cefd427e39d.tar.gz",
 )
 
-consensus_spec_version = "v1.5.0-alpha.6"
+consensus_spec_version = "v1.5.0-alpha.8"
 
 bls_test_version = "v0.1.1"
 
@@ -243,7 +243,7 @@ filegroup(
     visibility = ["//visibility:public"],
 )
     """,
-    integrity = "sha256-M7u/Ot/Vzorww+dFbHp0cxLyM2mezJjijCzq+LY3uvs=",
+    integrity = "sha256-BsGIbEyJuYrzhShGl0tHhR4lP5Qwno8R3k8a6YBR/DA=",
     url = "https://github.com/ethereum/consensus-spec-tests/releases/download/%s/general.tar.gz" % consensus_spec_version,
 )
 
@@ -259,7 +259,7 @@ filegroup(
     visibility = ["//visibility:public"],
 )
     """,
-    integrity = "sha256-deOSeLRsmHXvkRp8n2bs3HXdkGUJWWqu8KFM/QABbZg=",
+    integrity = "sha256-DkdvhPP2KiqUOpwFXQIFDCWCwsUDIC/xhTBD+TZevm0=",
     url = "https://github.com/ethereum/consensus-spec-tests/releases/download/%s/minimal.tar.gz" % consensus_spec_version,
 )
 
@@ -275,7 +275,7 @@ filegroup(
     visibility = ["//visibility:public"],
 )
     """,
-    integrity = "sha256-Zz7YCf6XVf57nzSEGq9ToflJFHM0lAGwhd18l9Rf3hA=",
+    integrity = "sha256-vkZqV0HB8A2Uc56C1Us/p5G57iaHL+zw2No93Xt6M/4=",
     url = "https://github.com/ethereum/consensus-spec-tests/releases/download/%s/mainnet.tar.gz" % consensus_spec_version,
 )
 
@@ -290,7 +290,7 @@ filegroup(
     visibility = ["//visibility:public"],
 )
     """,
-    integrity = "sha256-BoXckDxXnDcEmAjg/dQgf/tLiJsb6CT0aZvmWHFijrY=",
+    integrity = "sha256-D/HPAW61lKqjoWwl7N0XvhdX+67dCEFAy8JxVzqBGtU=",
     strip_prefix = "consensus-specs-" + consensus_spec_version[1:],
     url = "https://github.com/ethereum/consensus-specs/archive/refs/tags/%s.tar.gz" % consensus_spec_version,
 )

api/headers.go

@@ -1,5 +1,7 @@
 package api
 
+import "net/http"
+
 const (
 	VersionHeader                 = "Eth-Consensus-Version"
 	ExecutionPayloadBlindedHeader = "Eth-Execution-Payload-Blinded"
@@ -10,3 +12,9 @@ const (
 	EventStreamMediaType          = "text/event-stream"
 	KeepAlive                     = "keep-alive"
 )
+
+// SetSSEHeaders sets the headers needed for a server-sent event response.
+func SetSSEHeaders(w http.ResponseWriter) {
+	w.Header().Set("Content-Type", EventStreamMediaType)
+	w.Header().Set("Connection", KeepAlive)
+}

api/server/structs/BUILD.bazel

@@ -5,6 +5,7 @@ go_library(
     srcs = [
         "block.go",
         "conversions.go",
+        "conversions_blob.go",
         "conversions_block.go",
         "conversions_lightclient.go",
         "conversions_state.go",

api/server/structs/conversions.go

@@ -15,6 +15,7 @@ import (
 	"github.com/prysmaticlabs/prysm/v5/encoding/bytesutil"
 	"github.com/prysmaticlabs/prysm/v5/math"
 	enginev1 "github.com/prysmaticlabs/prysm/v5/proto/engine/v1"
+	ethv1 "github.com/prysmaticlabs/prysm/v5/proto/eth/v1"
 	eth "github.com/prysmaticlabs/prysm/v5/proto/prysm/v1alpha1"
 )
 
@@ -1475,12 +1476,15 @@ func DepositSnapshotFromConsensus(ds *eth.DepositSnapshot) *DepositSnapshot {
 	}
 }
 
-func PendingBalanceDepositsFromConsensus(ds []*eth.PendingBalanceDeposit) []*PendingBalanceDeposit {
-	deposits := make([]*PendingBalanceDeposit, len(ds))
+func PendingDepositsFromConsensus(ds []*eth.PendingDeposit) []*PendingDeposit {
+	deposits := make([]*PendingDeposit, len(ds))
 	for i, d := range ds {
-		deposits[i] = &PendingBalanceDeposit{
-			Index:  fmt.Sprintf("%d", d.Index),
-			Amount: fmt.Sprintf("%d", d.Amount),
+		deposits[i] = &PendingDeposit{
+			Pubkey:                hexutil.Encode(d.PublicKey),
+			WithdrawalCredentials: hexutil.Encode(d.WithdrawalCredentials),
+			Amount:                fmt.Sprintf("%d", d.Amount),
+			Signature:             hexutil.Encode(d.Signature),
+			Slot:                  fmt.Sprintf("%d", d.Slot),
 		}
 	}
 	return deposits
@@ -1508,3 +1512,37 @@ func PendingConsolidationsFromConsensus(cs []*eth.PendingConsolidation) []*Pendi
 	}
 	return consolidations
 }
+
+func HeadEventFromV1(event *ethv1.EventHead) *HeadEvent {
+	return &HeadEvent{
+		Slot:                      fmt.Sprintf("%d", event.Slot),
+		Block:                     hexutil.Encode(event.Block),
+		State:                     hexutil.Encode(event.State),
+		EpochTransition:           event.EpochTransition,
+		ExecutionOptimistic:       event.ExecutionOptimistic,
+		PreviousDutyDependentRoot: hexutil.Encode(event.PreviousDutyDependentRoot),
+		CurrentDutyDependentRoot:  hexutil.Encode(event.CurrentDutyDependentRoot),
+	}
+}
+
+func FinalizedCheckpointEventFromV1(event *ethv1.EventFinalizedCheckpoint) *FinalizedCheckpointEvent {
+	return &FinalizedCheckpointEvent{
+		Block:               hexutil.Encode(event.Block),
+		State:               hexutil.Encode(event.State),
+		Epoch:               fmt.Sprintf("%d", event.Epoch),
+		ExecutionOptimistic: event.ExecutionOptimistic,
+	}
+}
+
+func EventChainReorgFromV1(event *ethv1.EventChainReorg) *ChainReorgEvent {
+	return &ChainReorgEvent{
+		Slot:                fmt.Sprintf("%d", event.Slot),
+		Depth:               fmt.Sprintf("%d", event.Depth),
+		OldHeadBlock:        hexutil.Encode(event.OldHeadBlock),
+		NewHeadBlock:        hexutil.Encode(event.NewHeadBlock),
+		OldHeadState:        hexutil.Encode(event.OldHeadState),
+		NewHeadState:        hexutil.Encode(event.NewHeadState),
+		Epoch:               fmt.Sprintf("%d", event.Epoch),
+		ExecutionOptimistic: event.ExecutionOptimistic,
+	}
+}

api/server/structs/conversions_blob.go

@@ -0,0 +1,61 @@
+package structs
+
+import (
+	"strconv"
+
+	"github.com/prysmaticlabs/prysm/v5/api/server"
+	"github.com/prysmaticlabs/prysm/v5/encoding/bytesutil"
+	eth "github.com/prysmaticlabs/prysm/v5/proto/prysm/v1alpha1"
+)
+
+func (sc *Sidecar) ToConsensus() (*eth.BlobSidecar, error) {
+	if sc == nil {
+		return nil, errNilValue
+	}
+
+	index, err := strconv.ParseUint(sc.Index, 10, 64)
+	if err != nil {
+		return nil, server.NewDecodeError(err, "Index")
+	}
+
+	blob, err := bytesutil.DecodeHexWithLength(sc.Blob, 131072)
+	if err != nil {
+		return nil, server.NewDecodeError(err, "Blob")
+	}
+
+	kzgCommitment, err := bytesutil.DecodeHexWithLength(sc.KzgCommitment, 48)
+	if err != nil {
+		return nil, server.NewDecodeError(err, "KzgCommitment")
+	}
+
+	kzgProof, err := bytesutil.DecodeHexWithLength(sc.KzgProof, 48)
+	if err != nil {
+		return nil, server.NewDecodeError(err, "KzgProof")
+	}
+
+	header, err := sc.SignedBeaconBlockHeader.ToConsensus()
+	if err != nil {
+		return nil, server.NewDecodeError(err, "SignedBeaconBlockHeader")
+	}
+
+	// decode the commitment inclusion proof
+	var commitmentInclusionProof [][]byte
+	for _, proof := range sc.CommitmentInclusionProof {
+		proofBytes, err := bytesutil.DecodeHexWithLength(proof, 32)
+		if err != nil {
+			return nil, server.NewDecodeError(err, "CommitmentInclusionProof")
+		}
+		commitmentInclusionProof = append(commitmentInclusionProof, proofBytes)
+	}
+
+	bsc := &eth.BlobSidecar{
+		Index:                    index,
+		Blob:                     blob,
+		KzgCommitment:            kzgCommitment,
+		KzgProof:                 kzgProof,
+		SignedBlockHeader:        header,
+		CommitmentInclusionProof: commitmentInclusionProof,
+	}
+
+	return bsc, nil
+}

api/server/structs/conversions_block.go

@@ -20,6 +20,9 @@ import (
 var ErrUnsupportedConversion = errors.New("Could not determine api struct type to use for value")
 
 func (h *SignedBeaconBlockHeader) ToConsensus() (*eth.SignedBeaconBlockHeader, error) {
+	if h == nil {
+		return nil, errNilValue
+	}
 	msg, err := h.Message.ToConsensus()
 	if err != nil {
 		return nil, server.NewDecodeError(err, "Message")
@@ -36,6 +39,9 @@ func (h *SignedBeaconBlockHeader) ToConsensus() (*eth.SignedBeaconBlockHeader, e
 }
 
 func (h *BeaconBlockHeader) ToConsensus() (*eth.BeaconBlockHeader, error) {
+	if h == nil {
+		return nil, errNilValue
+	}
 	s, err := strconv.ParseUint(h.Slot, 10, 64)
 	if err != nil {
 		return nil, server.NewDecodeError(err, "Slot")
@@ -2548,6 +2554,8 @@ func SignedBeaconBlockMessageJsoner(block interfaces.ReadOnlySignedBeaconBlock)
 		return SignedBlindedBeaconBlockDenebFromConsensus(pbStruct)
 	case *eth.SignedBeaconBlockDeneb:
 		return SignedBeaconBlockDenebFromConsensus(pbStruct)
+	case *eth.SignedBlindedBeaconBlockElectra:
+		return SignedBlindedBeaconBlockElectraFromConsensus(pbStruct)
 	case *eth.SignedBeaconBlockElectra:
 		return SignedBeaconBlockElectraFromConsensus(pbStruct)
 	default:

api/server/structs/conversions_lightclient.go

@@ -84,6 +84,11 @@ func syncAggregateToJSON(input *v1.SyncAggregate) *SyncAggregate {
 }
 
 func lightClientHeaderContainerToJSON(container *v2.LightClientHeaderContainer) (json.RawMessage, error) {
+	// In the case that a finalizedHeader is nil.
+	if container == nil {
+		return nil, nil
+	}
+
 	beacon, err := container.GetBeacon()
 	if err != nil {
 		return nil, errors.Wrap(err, "could not get beacon block header")