Proposal for fixing issues in Access Node pt.3

[Guitarheroua]

Grant category

• Open source maintenance

Description

These tasks involve enhancements and optimizations to the Access and Execution Data APIs in the flow-go repository. They encompass a range of improvements, including optional event encoding, standardized JSON encoding, addition of heartbeat responses, and enabling grpc compression for more efficient communication between nodes. The tasks require a combination of code modification, algorithm implementation, benchmarking, and integration work to achieve the desired outcomes. Each task addresses specific aspects of API functionality and contributes to overall system performance and efficiency.

Proposed solution

1. [Access] Make CCF encoded events optional via request parameter | Fixes: [onflow/flow-go#4506]
   To implement the functionality requested in the issue, a list of endpoints from the AccessAPI and the ExecutionDataAPI should be modified with an additional parameter, which will specify the encoding for events, as currently it always uses CCF to JSON conversion from MessageToEventFromVersion. If not specified, default encoding should remain JSON. All related tests for modified endpoints should be adopted and extended to test new parameters.

2. [Access] Add heartbeat responses to event streaming API | Fixes: [onflow/flow-go#4215]
   To resolve the issue, the state_stream API ([SubscribeEvents](https://github.com/onflow/flow-go/blob/dafd8fd29c476cacc497f79940412ba377f723fe/engine/access/state_stream/handler.go#L105) call) should be expanded by adding an additional option parameter to specify the heartbeat interval. The [EventsBackend](https://github.com/onflow/flow-go/blob/dafd8fd29c476cacc497f79940412ba377f723fe/engine/access/state_stream/backend_events.go) should be upgraded by returning periodic EventsResponse with the last block id, height, and empty events when no events are found from such block interval. Existing [backend_events_test.go](https://github.com/onflow/flow-go/blob/dafd8fd29c476cacc497f79940412ba377f723fe/engine/access/state_stream/backend_events.go) tests should be upgraded and new unit tests for event streaming should be added.

3. [Access] Enable grpc compression | Fixes: [onflow/flow-go#4503]

• Research compression and how it can be integrated to access/execution nodes communication, to compress/decompress request/response data. We will consider the 4 most likely suited algorithms for this task: gzip, DEFLATE, LZ4, Snappy.
• Implement those compressing algorithms ( gzip already implemented so + 3 new ones).
• Implement a synthetic benchmark to measure the compression of those algorithms. Compare to choose the most efficient algorithm.
• Prepare comparison doc, discuss with the team, and decide on the best option.
• Integrate, and make mandatory compressing algorithm for access/execution nodes.

Impact

Proposed fixes will improve Access Node functionality's specific aspects and contribute to overall system performance and efficiency.

Milestones and funding

Milestone 1

Milestone	Deliverables	Timeline	Risks	USD proposal
1 - [Access] Make CCF encoded events optional via request parameter	Fixes: onflow/flow-go#4506	96h	6240	$
2 - [Access] Add heartbeat responses to event streaming API	Fixes: onflow/flow-go#4215	72h	4680	$

Milestone 2

Milestone	Deliverables	Timeline	Risks	USD proposal
3 - [Access] Enable grpc compression	Fixes: onflow/flow-go#4503	132h	8580	$

Total funding proposed(USD): 19500

Team

Name	Role	Bio	Contact
Andrii	Blockchain Engineer	Software Engineer with over 8 years of technical experience. Primary skills include C++(Standards 11,14,17), Rust, and Golang. I have been highly focused on blockchain development for the last few years. Here are a few examples of my projects that could be interesting when applying for this grant: - 2016-2019: Supply chain planning and management tools/applications for huge manufacturers and logistic businesses. Worked in a team of 20+ engineers. Gained experience in Golang. Build integrations between applications using REST API. - 2021-2022: Complex DeFi lending/borrowing system using Rust/Solana. Smart contracts design, implementation, and testing (using typescript). Project successfully works on the market and ranks in the top 100 of Coinmarketcap. - 2022-2023: Development of Domain name service using Rust on Elrond blockchain. Smart contracts / Backend / deployment.	[email protected]
Uliana	Blockchain Engineer	Software Engineer with over 5 years of technical experience. Primary skills include C++(Standards 11,14,17), Rust, and Golang. I have been highly focused on blockchain development from the very beginning of my career, here are a few examples of my projects: - 2018-2021: Creation of a multi-currency light wallet, Lightning Network-ready, that features its own DEX. C++ Development of blockchain specific features, creation of custom version of private lightning network using Golang. - 2021-2022: Complex DeFi lending/borrowing system using Rust/Solana. Smart contracts design, implementation, and testing (using typescript). - 2021-2023: Developing smart contracts for NFT Marketplace platform startup using Rust and Cosmos.	[email protected]