eds: Blockstore provided by the Store should support Put

[Wondertan]

Context

During the design and implementation of #1099, we missed the necessity of caching the inner and leaf nodes during reconstruction via the Blockservice, leading to Bitswap having no access to them. This is due to all the Puts being discarded on the Blockstore.

Problems

• FNs do not share proofs and data during reconstruction between each other.
• FNs cannot generate BEFPs for withheld blocks.

Solution

The proposed solution here is changing Blockstore to support Put operations:

• Blockstore over on-disk Datastore

  • Dagstore's Blockstore would write to disk and, on reads, check the index first and then the Datastore.
  • The default badger Datastore scales well for write-heavy loads, and considering sparse read patterns for reconstruction and BEFP, it should also suit read requirements.
  • The only complication is clean-up, which can be naively solved by iterating over each stored piece with the DeleteBlock method on Blockstore at the end of the reconstruction session(or even after we put an indexed eds to avoid interruption of serving data to other nodes over bitswap)

• Blockstore over in-mem Datastore

  • Similar cleanup complication.
  • Similar implementation time estimate as for on-disk
  • Less robust. Does not tolerates restarts.

Testing

We have a (flaky) Swamp reconstruction test that asserts that a FN can reconstruct a block from light nodes only, but we also need to add a test similar to the test below, which validates that FNs can collectively rebuild the block.

celestia-node/share/availability/full/reconstruction_test.go

Line 162 in 4d98694

func TestShareAvailable_DisconnectedFullNodes(t *testing.T) {

Once we know that this works fine in Swamp, we should also replicate the same scenario on TG.

[musalbas]

I assume that Put would only be called if shrexeds fails to get the full ODS, thus falling back to sampling anyway, right?

[Wondertan]

@musalbas, right. The sampling/reconstruction is started once we cannot get ODS after a while.