Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: remoteDb only delay syncs for specified peers #130

Open
wants to merge 65 commits into
base: state_expiry_mvp0.1_dev
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
fc19bf5
flags: add state expiry enable flag;
0xbundler Aug 18, 2023
e146978
feat: add meta info
asyukii Aug 18, 2023
29adf96
feat: add path proof generation
asyukii Aug 21, 2023
887dfdc
feat: add revive operation
asyukii Aug 21, 2023
3a98bb6
state/snapshot: support snapshot prune, epoch meta;
0xbundler Aug 24, 2023
48859f7
state/snapshot: support get/put in statedb;
0xbundler Aug 24, 2023
abed248
feat: implement epoch-based trie
asyukii Aug 23, 2023
7ce6beb
feat: add eth_getStorageReviveProof
asyukii Aug 24, 2023
d700e95
ethdb/fullstatedb: add FullStateDB interface;
0xbundler Aug 28, 2023
48d0784
core/state: implement state object revive storage trie
asyukii Aug 29, 2023
f0435dd
core/types: modify metadata type
asyukii Aug 29, 2023
b393825
core/state, trie: access trie updates with current epoch
asyukii Aug 29, 2023
2c7c459
feat: add trie prune logic
asyukii Aug 25, 2023
22fc2ab
trie/epochmeta: add trie epoch meta storage;
0xbundler Aug 28, 2023
19fd26c
trie/committer: support epoch meta commit;
0xbundler Aug 29, 2023
2350ed4
pruner: support prune expired trie/snapshot kv;
0xbundler Aug 30, 2023
cca3319
pruner: support prune expired trie/snapshot kv;
0xbundler Aug 30, 2023
b8ce056
state/trieprefetcher: support handle expired state;
0xbundler Sep 1, 2023
794273c
fix: revive storage trie from remoteDB error
asyukii Sep 1, 2023
7633607
trie/proof: fix child hash validate bug;
0xbundler Sep 4, 2023
a6e4e30
fix: revive trie use nubs method
asyukii Sep 4, 2023
58e80ab
core/state: fix revive error, trie prefetcher copy bug;
0xbundler Sep 5, 2023
e0de4ac
logs: opt some state expiry logs level to debug;
0xbundler Sep 7, 2023
753658c
trie/trie: fix compile error;
0xbundler Sep 18, 2023
0c86308
bugfix: fix some proof generate, revive bugs;
0xbundler Sep 8, 2023
001a2c5
trie/epochmeta: opt graceful shutdown logic;
0xbundler Sep 11, 2023
079a1c6
trie/inspect: add inspect trie tools;
0xbundler Sep 11, 2023
7ac0a0b
metrics: opt some timer, add more miner metrics;
0xbundler Sep 12, 2023
c9bd5b4
fixbugs: fix prune initial;
0xbundler Sep 12, 2023
91577fb
bugfix: fix trie update;
0xbundler Sep 20, 2023
16f91a8
bugfix: fix state prefetcher concurrent bugs;
0xbundler Sep 22, 2023
4b78013
fix: rebase bsc/pbss_active conflicts
asyukii Sep 25, 2023
26e7967
trie/proof: fix pbss trie proof generate bug;
0xbundler Sep 26, 2023
7dde14a
pruner: support PBSS expired state prune;
0xbundler Sep 26, 2023
85db375
rpc(revive): get from latest block num
asyukii Sep 26, 2023
43dde20
core/state, trie: fix bad block
asyukii Sep 26, 2023
c05a729
feat: insert new slot in snapshot, prevent execution touch trie;
0xbundler Sep 25, 2023
e4235be
core/state: subfetcher revive state in batch mode
asyukii Sep 27, 2023
634d117
trie: add unit tests
asyukii Sep 27, 2023
d9e7568
api: add cache for storage revive proof
asyukii Oct 2, 2023
2bc3c9c
chore(cmd/geth): enable state expiry only with PBSS
asyukii Oct 3, 2023
c3a31d7
pruner: opt size statistic;
0xbundler Oct 8, 2023
bdc2fc9
flags: refactor state expiry config;
0xbundler Oct 9, 2023
b9a07a1
feat(snap): add state expiry support to snap sync
asyukii Oct 5, 2023
0772f61
pruner: fix some prune bugs;
0xbundler Oct 10, 2023
2032bfa
trie/epochmeta: add bloom filter, opt lock;
0xbundler Oct 11, 2023
5f776c2
metrics: add some trace metrics for epoch meta;
0xbundler Oct 12, 2023
4b05dc4
core/state: fix snapshot recovery
asyukii Oct 12, 2023
4e7a438
trie: fix rebase error;
0xbundler Oct 16, 2023
811f37a
trie: opt trie update;
0xbundler Oct 16, 2023
7a7abb9
pruner: opt expired prune, add more logs;
0xbundler Oct 17, 2023
5a5f371
state: add some revive metrics;
0xbundler Oct 17, 2023
74169f7
fix: revert to master change;
0xbundler Oct 17, 2023
e98a619
trie/pathdb: support account meta/epoch meta query;
0xbundler Oct 18, 2023
a1a0d4d
fix: missing epochMaps init
asyukii Oct 18, 2023
d2ad0ec
trie/pathdb: fix clean cache low hit rate issue;
0xbundler Oct 20, 2023
272522b
prune: add more concurrent logics;
0xbundler Oct 22, 2023
7fb891c
trie/typednode: opt encode/decode performance;
0xbundler Oct 23, 2023
c502191
state/state_object: add more expired metrics;
0xbundler Oct 24, 2023
749bcc2
eth/downloader: state expiry remotedb keep behind the latest;
0xbundler Oct 25, 2023
d24d379
ethapi: opt storage proof logic;
0xbundler Oct 26, 2023
52d8606
state/stateobject: add dirtystorage touch & revive logic;
0xbundler Oct 30, 2023
40a6e30
state/stateobject: fix revive in diff & delete shrink miss node bug;
0xbundler Nov 1, 2023
128eb98
ut: fix some broken ut;
0xbundler Nov 3, 2023
68a24ca
feat: remoteDb only delay syncs for specified peers
asyukii Nov 6, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/commit-lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,13 @@ on:
branches:
- master
- develop
- state_expiry_mvp0.1_dev

pull_request:
branches:
- master
- develop
- state_expiry_mvp0.1_dev

jobs:
commitlint:
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/integration-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,13 @@ on:
branches:
- master
- develop
- state_expiry_mvp0.1_dev

pull_request:
branches:
- master
- develop
- state_expiry_mvp0.1_dev

jobs:
truffle-test:
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,13 @@ on:
branches:
- master
- develop
- state_expiry_mvp0.1_dev

pull_request:
branches:
- master
- develop
- state_expiry_mvp0.1_dev

jobs:
golang-lint:
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/unit-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,13 @@ on:
branches:
- master
- develop
- state_expiry_mvp0.1_dev

pull_request:
branches:
- master
- develop
- state_expiry_mvp0.1_dev

jobs:
unit-test:
Expand Down
2 changes: 1 addition & 1 deletion accounts/abi/bind/backends/simulated.go
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@ func (b *SimulatedBackend) stateByBlockNumber(ctx context.Context, blockNumber *
if err != nil {
return nil, err
}
return b.blockchain.StateAt(block.Root())
return b.blockchain.StateAt(block.Root(), block.Hash(), block.Number())
}

// CodeAt returns the code associated with a certain account in the blockchain.
Expand Down
14 changes: 7 additions & 7 deletions cmd/geth/chaincmd.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ var (
Flags: flags.Merge([]cli.Flag{
utils.CachePreimagesFlag,
utils.StateSchemeFlag,
}, utils.DatabasePathFlags),
}, utils.DatabasePathFlags, utils.StateExpiryBaseFlags),
Description: `
The init command initializes a new genesis block and definition for the network.
This is a destructive action and changes the network in which you will be
Expand Down Expand Up @@ -86,7 +86,7 @@ It expects the genesis file as argument.`,
Name: "dumpgenesis",
Usage: "Dumps genesis block JSON configuration to stdout",
ArgsUsage: "",
Flags: append([]cli.Flag{utils.DataDirFlag}, utils.NetworkFlags...),
Flags: flags.Merge([]cli.Flag{utils.DataDirFlag}, utils.NetworkFlags, utils.StateExpiryBaseFlags),
Description: `
The dumpgenesis command prints the genesis configuration of the network preset
if one is set. Otherwise it prints the genesis from the datadir.`,
Expand Down Expand Up @@ -121,7 +121,7 @@ if one is set. Otherwise it prints the genesis from the datadir.`,
utils.TransactionHistoryFlag,
utils.StateSchemeFlag,
utils.StateHistoryFlag,
}, utils.DatabasePathFlags),
}, utils.DatabasePathFlags, utils.StateExpiryBaseFlags),
Description: `
The import command imports blocks from an RLP-encoded form. The form can be one file
with several RLP-encoded blocks, or several files can be used.
Expand All @@ -138,7 +138,7 @@ processing will proceed even if an individual RLP-file import failure occurs.`,
utils.CacheFlag,
utils.SyncModeFlag,
utils.StateSchemeFlag,
}, utils.DatabasePathFlags),
}, utils.DatabasePathFlags, utils.StateExpiryBaseFlags),
Description: `
Requires a first argument of the file to write to.
Optional second and third arguments control the first and
Expand All @@ -154,7 +154,7 @@ be gzipped.`,
Flags: flags.Merge([]cli.Flag{
utils.CacheFlag,
utils.SyncModeFlag,
}, utils.DatabasePathFlags),
}, utils.DatabasePathFlags, utils.StateExpiryBaseFlags),
Description: `
The import-preimages command imports hash preimages from an RLP encoded stream.
It's deprecated, please use "geth db import" instead.
Expand All @@ -168,7 +168,7 @@ It's deprecated, please use "geth db import" instead.
Flags: flags.Merge([]cli.Flag{
utils.CacheFlag,
utils.SyncModeFlag,
}, utils.DatabasePathFlags),
}, utils.DatabasePathFlags, utils.StateExpiryBaseFlags),
Description: `
The export-preimages command exports hash preimages to an RLP encoded stream.
It's deprecated, please use "geth db export" instead.
Expand All @@ -188,7 +188,7 @@ It's deprecated, please use "geth db export" instead.
utils.StartKeyFlag,
utils.DumpLimitFlag,
utils.StateSchemeFlag,
}, utils.DatabasePathFlags),
}, utils.DatabasePathFlags, utils.StateExpiryBaseFlags),
Description: `
This command dumps out the state for a given block (or latest, if none provided).
`,
Expand Down
2 changes: 1 addition & 1 deletion cmd/geth/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ var (
Name: "dumpconfig",
Usage: "Export configuration values in a TOML format",
ArgsUsage: "<dumpfile (optional)>",
Flags: flags.Merge(nodeFlags, rpcFlags),
Flags: flags.Merge(nodeFlags, rpcFlags, utils.StateExpiryBaseFlags),
Description: `Export configuration values in TOML format (to stdout by default).`,
}

Expand Down
2 changes: 1 addition & 1 deletion cmd/geth/consolecmd.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ var (
Action: localConsole,
Name: "console",
Usage: "Start an interactive JavaScript environment",
Flags: flags.Merge(nodeFlags, rpcFlags, consoleFlags),
Flags: flags.Merge(nodeFlags, rpcFlags, consoleFlags, utils.StateExpiryBaseFlags),
Description: `
The Geth console is an interactive shell for the JavaScript runtime environment
which exposes a node admin interface as well as the Ðapp JavaScript API.
Expand Down
118 changes: 102 additions & 16 deletions cmd/geth/dbcmd.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ package main

import (
"bytes"
"errors"
"fmt"
"math"
"os"
Expand Down Expand Up @@ -49,7 +50,7 @@ var (
Name: "removedb",
Usage: "Remove blockchain and state databases",
ArgsUsage: "",
Flags: utils.DatabasePathFlags,
Flags: flags.Merge(utils.DatabasePathFlags, utils.StateExpiryBaseFlags),
Description: `
Remove blockchain and state databases`,
}
Expand All @@ -59,6 +60,7 @@ Remove blockchain and state databases`,
ArgsUsage: "",
Subcommands: []*cli.Command{
dbInspectCmd,
dbInspectTrieCmd,
dbStatCmd,
dbCompactCmd,
dbGetCmd,
Expand All @@ -84,15 +86,25 @@ Remove blockchain and state databases`,
ArgsUsage: "<prefix> <start>",
Flags: flags.Merge([]cli.Flag{
utils.SyncModeFlag,
}, utils.NetworkFlags, utils.DatabasePathFlags),
}, utils.NetworkFlags, utils.DatabasePathFlags, utils.StateExpiryBaseFlags),
Usage: "Inspect the storage size for each type of data in the database",
Description: `This commands iterates the entire database. If the optional 'prefix' and 'start' arguments are provided, then the iteration is limited to the given subset of data.`,
}
dbInspectTrieCmd = &cli.Command{
Action: inspectTrie,
Name: "inspect-trie",
ArgsUsage: "<blocknum> <jobnum>",
Flags: flags.Merge([]cli.Flag{
utils.SyncModeFlag,
}, utils.DatabasePathFlags, utils.StateExpiryBaseFlags),
Usage: "Inspect the MPT tree of the account and contract.",
Description: `This commands iterates the entrie WorldState.`,
}
dbCheckStateContentCmd = &cli.Command{
Action: checkStateContent,
Name: "check-state-content",
ArgsUsage: "<start (optional)>",
Flags: flags.Merge(utils.NetworkFlags, utils.DatabasePathFlags),
Flags: flags.Merge(utils.NetworkFlags, utils.DatabasePathFlags, utils.StateExpiryBaseFlags),
Usage: "Verify that state data is cryptographically correct",
Description: `This command iterates the entire database for 32-byte keys, looking for rlp-encoded trie nodes.
For each trie node encountered, it checks that the key corresponds to the keccak256(value). If this is not true, this indicates
Expand Down Expand Up @@ -143,7 +155,7 @@ a data corruption.`,
Usage: "Print leveldb statistics",
Flags: flags.Merge([]cli.Flag{
utils.SyncModeFlag,
}, utils.NetworkFlags, utils.DatabasePathFlags),
}, utils.NetworkFlags, utils.DatabasePathFlags, utils.StateExpiryBaseFlags),
}
dbCompactCmd = &cli.Command{
Action: dbCompact,
Expand All @@ -153,7 +165,7 @@ a data corruption.`,
utils.SyncModeFlag,
utils.CacheFlag,
utils.CacheDatabaseFlag,
}, utils.NetworkFlags, utils.DatabasePathFlags),
}, utils.NetworkFlags, utils.DatabasePathFlags, utils.StateExpiryBaseFlags),
Description: `This command performs a database compaction.
WARNING: This operation may take a very long time to finish, and may cause database
corruption if it is aborted during execution'!`,
Expand All @@ -165,7 +177,7 @@ corruption if it is aborted during execution'!`,
ArgsUsage: "<hex-encoded key>",
Flags: flags.Merge([]cli.Flag{
utils.SyncModeFlag,
}, utils.NetworkFlags, utils.DatabasePathFlags),
}, utils.NetworkFlags, utils.DatabasePathFlags, utils.StateExpiryBaseFlags),
Description: "This command looks up the specified database key from the database.",
}
dbDeleteCmd = &cli.Command{
Expand All @@ -175,7 +187,7 @@ corruption if it is aborted during execution'!`,
ArgsUsage: "<hex-encoded key>",
Flags: flags.Merge([]cli.Flag{
utils.SyncModeFlag,
}, utils.NetworkFlags, utils.DatabasePathFlags),
}, utils.NetworkFlags, utils.DatabasePathFlags, utils.StateExpiryBaseFlags),
Description: `This command deletes the specified database key from the database.
WARNING: This is a low-level operation which may cause database corruption!`,
}
Expand All @@ -186,7 +198,7 @@ WARNING: This is a low-level operation which may cause database corruption!`,
ArgsUsage: "<hex-encoded key> <hex-encoded value>",
Flags: flags.Merge([]cli.Flag{
utils.SyncModeFlag,
}, utils.NetworkFlags, utils.DatabasePathFlags),
}, utils.NetworkFlags, utils.DatabasePathFlags, utils.StateExpiryBaseFlags),
Description: `This command sets a given database key to the given value.
WARNING: This is a low-level operation which may cause database corruption!`,
}
Expand All @@ -198,7 +210,7 @@ WARNING: This is a low-level operation which may cause database corruption!`,
Flags: flags.Merge([]cli.Flag{
utils.SyncModeFlag,
utils.StateSchemeFlag,
}, utils.NetworkFlags, utils.DatabasePathFlags),
}, utils.NetworkFlags, utils.DatabasePathFlags, utils.StateExpiryBaseFlags),
Description: "This command looks up the specified database key from the database.",
}
dbDumpFreezerIndex = &cli.Command{
Expand All @@ -208,7 +220,7 @@ WARNING: This is a low-level operation which may cause database corruption!`,
ArgsUsage: "<freezer-type> <table-type> <start (int)> <end (int)>",
Flags: flags.Merge([]cli.Flag{
utils.SyncModeFlag,
}, utils.NetworkFlags, utils.DatabasePathFlags),
}, utils.NetworkFlags, utils.DatabasePathFlags, utils.StateExpiryBaseFlags),
Description: "This command displays information about the freezer index.",
}
dbImportCmd = &cli.Command{
Expand All @@ -218,7 +230,7 @@ WARNING: This is a low-level operation which may cause database corruption!`,
ArgsUsage: "<dumpfile> <start (optional)",
Flags: flags.Merge([]cli.Flag{
utils.SyncModeFlag,
}, utils.NetworkFlags, utils.DatabasePathFlags),
}, utils.NetworkFlags, utils.DatabasePathFlags, utils.StateExpiryBaseFlags),
Description: "The import command imports the specific chain data from an RLP encoded stream.",
}
dbExportCmd = &cli.Command{
Expand All @@ -228,7 +240,7 @@ WARNING: This is a low-level operation which may cause database corruption!`,
ArgsUsage: "<type> <dumpfile>",
Flags: flags.Merge([]cli.Flag{
utils.SyncModeFlag,
}, utils.NetworkFlags, utils.DatabasePathFlags),
}, utils.NetworkFlags, utils.DatabasePathFlags, utils.StateExpiryBaseFlags),
Description: "Exports the specified chain data to an RLP encoded stream, optionally gzip-compressed.",
}
dbMetadataCmd = &cli.Command{
Expand All @@ -237,15 +249,16 @@ WARNING: This is a low-level operation which may cause database corruption!`,
Usage: "Shows metadata about the chain status.",
Flags: flags.Merge([]cli.Flag{
utils.SyncModeFlag,
}, utils.NetworkFlags, utils.DatabasePathFlags),
}, utils.NetworkFlags, utils.DatabasePathFlags, utils.StateExpiryBaseFlags),
Description: "Shows metadata about the chain status.",
}
ancientInspectCmd = &cli.Command{
Action: ancientInspect,
Name: "inspect-reserved-oldest-blocks",
Flags: []cli.Flag{
utils.DataDirFlag,
},
Flags: flags.Merge(
[]cli.Flag{utils.DataDirFlag},
utils.StateExpiryBaseFlags,
),
Usage: "Inspect the ancientStore information",
Description: `This commands will read current offset from kvdb, which is the current offset and starting BlockNumber
of ancientStore, will also displays the reserved number of blocks in ancientStore `,
Expand Down Expand Up @@ -352,6 +365,79 @@ func ancientInspect(ctx *cli.Context) error {
return rawdb.AncientInspect(db)
}

func inspectTrie(ctx *cli.Context) error {
if ctx.NArg() < 1 {
return fmt.Errorf("required arguments: %v", ctx.Command.ArgsUsage)
}

if ctx.NArg() > 3 {
return fmt.Errorf("Max 3 arguments: %v", ctx.Command.ArgsUsage)
}

var (
blockNumber uint64
blockRoot common.Hash
jobnum uint64
)

stack, _ := makeConfigNode(ctx)
defer stack.Close()

db := utils.MakeChainDatabase(ctx, stack, true, true)
defer db.Close()

if ctx.NArg() >= 1 {
if ctx.Args().Get(0) == "latest" {
headBlock := rawdb.ReadHeadBlock(db)
if headBlock == nil {
return errors.New("failed to load head block")
}
blockNumber = headBlock.NumberU64()
blockRoot = headBlock.Root()
} else if ctx.Args().Get(0) == "snapshot" {
blockRoot = rawdb.ReadSnapshotRoot(db)
blockNumber = math.MaxUint64
} else {
var err error
blockNumber, err = strconv.ParseUint(ctx.Args().Get(0), 10, 64)
if err != nil {
return fmt.Errorf("failed to Parse blocknum, Args[0]: %v, err: %v", ctx.Args().Get(0), err)
}
blockHash := rawdb.ReadCanonicalHash(db, blockNumber)
block := rawdb.ReadBlock(db, blockHash, blockNumber)
blockRoot = block.Root()
}

if ctx.NArg() == 1 {
jobnum = 1000
} else {
var err error
jobnum, err = strconv.ParseUint(ctx.Args().Get(1), 10, 64)
if err != nil {
return fmt.Errorf("failed to Parse jobnum, Args[1]: %v, err: %v", ctx.Args().Get(1), err)
}
}

if (blockRoot == common.Hash{}) {
log.Error("Empty root hash")
}
fmt.Printf("ReadBlockHeader, root: %v, blocknum: %v\n", blockRoot, blockNumber)
trieDB := trie.NewDatabase(db, nil)
theTrie, err := trie.New(trie.TrieID(blockRoot), trieDB)
if err != nil {
fmt.Printf("fail to new trie tree, err: %v, rootHash: %v\n", err, blockRoot.String())
return err
}
theInspect, err := trie.NewInspector(trieDB, theTrie, blockNumber, jobnum)
if err != nil {
return err
}
theInspect.Run()
theInspect.DisplayResult()
}
return nil
}

func checkStateContent(ctx *cli.Context) error {
var (
prefix []byte
Expand Down
Loading
Loading