Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Runtime: Module to Migrate Built-Ins to Core BPF #8

Closed
wants to merge 49 commits into from

Conversation

buffalojoec
Copy link
Owner

I'm introducing this PR as a draft to hopefully gather some feedback on the
approach and some of the various things this migration path touches.

I've carefully structured the commits to ensure maximum readability, including
saving the Bank modifications required for testing the migrations for last.


Problem

Now that SIMD 0088
has been merged, the runtime requires a code path for migrating built-in
programs to Core BPF.

Solution

The static BUILTINS list, which houses the source-of-truth for all runtime
built-in programs, contains a feature_id field for activating new built-in
programs.

Building on this architecture, I've included a new field for
core_bpf_migration configurations, which the runtime can use on epoch rollover
to migrate a built-in to Core BPF and ensure it's handled properly by both the
Bank and the program cache.

The config defines two main fields:

  • Source BPF Program: The upgradeable BPF program that will be moved in place of
    the built-in.
  • Feature ID: The feature ID whose activation will trigger the migration.

To ensure maximum safety when enabling migrations, I've abstracted all of the
checks and account manipulation required to perform the migration away into the
builtin module's sub-module core_bpf_migration.

To migrate a built-in program to Core BPF, contributors need only add the source
program ID and the feature ID to the built-in's config.

@buffalojoec buffalojoec force-pushed the migrate-builtin-module branch 2 times, most recently from f923aec to f817190 Compare February 8, 2024 02:11
pgarg66 and others added 28 commits February 8, 2024 15:32
The commit adds should_chain_merkle_shreds to incrementally roll out
chained Merkle shreds to clusters.
…s#34925)

- Add mode to output individual pubkeys
- Add mode to output program accounts
* build(deps): bump serde_yaml from 0.9.30 to 0.9.31

Bumps [serde_yaml](https://github.com/dtolnay/serde-yaml) from 0.9.30 to 0.9.31.
- [Release notes](https://github.com/dtolnay/serde-yaml/releases)
- [Commits](dtolnay/serde-yaml@0.9.30...0.9.31)

---
updated-dependencies:
- dependency-name: serde_yaml
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>
* build(deps): bump num-derive from 0.4.1 to 0.4.2

Bumps [num-derive](https://github.com/rust-num/num-derive) from 0.4.1 to 0.4.2.
- [Changelog](https://github.com/rust-num/num-derive/blob/master/RELEASES.md)
- [Commits](rust-num/num-derive@num-derive-0.4.1...num-derive-0.4.2)

---
updated-dependencies:
- dependency-name: num-derive
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>
Bumps [hidapi](https://github.com/ruabmbua/hidapi-rs) from 2.5.1 to 2.6.0.
- [Release notes](https://github.com/ruabmbua/hidapi-rs/releases)
- [Commits](ruabmbua/hidapi-rs@v2.5.1...v2.6.0)

---
updated-dependencies:
- dependency-name: hidapi
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps): bump num-traits from 0.2.17 to 0.2.18

Bumps [num-traits](https://github.com/rust-num/num-traits) from 0.2.17 to 0.2.18.
- [Changelog](https://github.com/rust-num/num-traits/blob/master/RELEASES.md)
- [Commits](rust-num/num-traits@num-traits-0.2.17...num-traits-0.2.18)

---
updated-dependencies:
- dependency-name: num-traits
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>
* chore: update crowdin and add serve command

* feat: upload all desired files to crowdin

* fix: absolute urls for pdfs

* fix: do not import components with relative paths

* feat: updated readme

* fix: whitespace
…olana-labs#35065)

#### Problem
There're some test functions that have been used in different
mod in TieredStorage.  It's better to have one same place for
all tiere-storage related test functions.

#### Summary of Changes
Created test_utils.rs under /tiered_storage and move test-related
functions into it.

#### Test Plan
Existing tests.
* Update SPL dependency versions for 2.0 bump

* Update Cargo.lock

* Update token-2022 version to v2.0.1

* pin spl-instruction-padding 0.1.1 and spl-tlv-account-resolution 0.5.2

* pin programs/sbf spl-tlv-account-resolution 0.5.2
* build(deps): bump bytemuck from 1.14.2 to 1.14.3

Bumps [bytemuck](https://github.com/Lokathor/bytemuck) from 1.14.2 to 1.14.3.
- [Changelog](https://github.com/Lokathor/bytemuck/blob/main/changelog.md)
- [Commits](Lokathor/bytemuck@v1.14.2...v1.14.3)

---
updated-dependencies:
- dependency-name: bytemuck
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>
* build(deps): bump chrono from 0.4.33 to 0.4.34

Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.33 to 0.4.34.
- [Release notes](https://github.com/chronotope/chrono/releases)
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md)
- [Commits](chronotope/chrono@v0.4.33...v0.4.34)

---
updated-dependencies:
- dependency-name: chrono
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>
* build(deps): bump thiserror from 1.0.56 to 1.0.57

Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.56 to 1.0.57.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](dtolnay/thiserror@1.0.56...1.0.57)

---
updated-dependencies:
- dependency-name: thiserror
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>
* build(deps): bump indexmap from 2.2.2 to 2.2.3

Bumps [indexmap](https://github.com/indexmap-rs/indexmap) from 2.2.2 to 2.2.3.
- [Changelog](https://github.com/indexmap-rs/indexmap/blob/master/RELEASES.md)
- [Commits](indexmap-rs/indexmap@2.2.2...2.2.3)

---
updated-dependencies:
- dependency-name: indexmap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>
* build(deps): bump either from 1.9.0 to 1.10.0

Bumps [either](https://github.com/rayon-rs/either) from 1.9.0 to 1.10.0.
- [Commits](rayon-rs/either@1.9.0...1.10.0)

---
updated-dependencies:
- dependency-name: either
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>
* build(deps): bump ahash from 0.8.7 to 0.8.8

Bumps [ahash](https://github.com/tkaitchuck/ahash) from 0.8.7 to 0.8.8.
- [Release notes](https://github.com/tkaitchuck/ahash/releases)
- [Commits](tkaitchuck/aHash@0.8.7...v0.8.8)

---
updated-dependencies:
- dependency-name: ahash
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>
* build(deps): bump indicatif from 0.17.7 to 0.17.8

Bumps [indicatif](https://github.com/console-rs/indicatif) from 0.17.7 to 0.17.8.
- [Release notes](https://github.com/console-rs/indicatif/releases)
- [Commits](https://github.com/console-rs/indicatif/commits)

---
updated-dependencies:
- dependency-name: indicatif
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>
…a-labs#35158)

The directory is currently named with the expected_shred_version;
however, the backup contains shreds that do NOT match the
expected_shred_version. So, use the found (incorrect) shred version in
the name instead.
* Replace LoadedPrograms::replenish() with LoadedPrograms::assign_program().

* Removes LoadedPrograms::replenish().

* Defines replacement by having the same loaded program type.

* Implements a proper insertion sort with a two key comparison operator.
Lichtso and others added 21 commits February 14, 2024 17:28
…na-labs#35146)

* Adds check that only loaded programs can be unloaded.

* Removes unused code.

* Adds test_unloaded().
…et subcommand (solana-labs#35200)

make LOOKUP_TABLE_ADDRESS arg required for address-lookup-table get subcommand
* Cleans up disable_bpf_loader_instructions.

* fix test_program_sbf_disguised_as_sbf_loader

* remove bpf loader execute bench

* Revert "remove bpf loader execute bench"

This reverts commit f3042ee.

* move test utility functions out of test file

* update bench to loader v3

* clippy

* fix dev-context build

* fix dev-context import

* dev-context-util

* move dev-context-util attr to module level for loader_utils

---------

Co-authored-by: HaoranYi <[email protected]>
…olana-labs#35207)

Already covered by other metrics like the filtered retryable and the
number filtered.
…pshot (solana-labs#35155)

* Add warn log if capitalization changes during create-snapshot

* Add enable-capitalization-change flag

* Print capitalization message at end
The typo in the function names convert_edianness_64 and convert_edianness_128 has been corrected to convert_endianness_64 and convert_endianness_128 respectively.
* Update README.md

grammar fixed

* Update docs/README.md

---------

Co-authored-by: Tyera <[email protected]>
…ana-labs#35143)

#### Problem
While accounts-db might not invoke appends_account twice
for the same AccountsFile, TieredStorage::write_accounts()
itself isn't thread-safe, and it depends on the above accounts-db
assumption.

#### Summary of Changes
This PR makes TieredStorage::write_accounts() thread-safe.
So only the first thread that successfully updates the already_written
flag can proceed and write the input accounts.  All subsequent
calls to write_accounts() will be a no-op and return AttemptToUpdateReadOnly
Error.
@buffalojoec buffalojoec force-pushed the migrate-builtin-module branch from f817190 to f6af900 Compare February 19, 2024 04:30
buffalojoec pushed a commit that referenced this pull request Mar 5, 2024
* ci: fix windows build

* ci: publish sdk docker image with the new name

* update automerge status
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.