Skip to content

Commit

Permalink
Merge pull request #355 from lidofinance/feat/oracle-v3-fixes-4
Browse files Browse the repository at this point in the history 
Feat/oracle v3 fixes 4
  • Loading branch information
@F4ever
F4ever authored Apr 14, 2023
2 parents 7c22b10 + 8842852 commit 0661a89
Show file tree
Hide file tree
Showing 20 changed files with 653 additions and 119 deletions.
45 changes: 24 additions & 21 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ Full variables list could be found [here](https://github.com/lidofinance/lido-or
Set required values. It will be enough to run the oracle in _check mode_.
2. Check that your environment is ready to run the oracle using the following command:
```bash
docker run --env-file .env --rm lidofinance/oracle:{tag} check
docker run -ti --env-file .env --rm lidofinance/oracle:{tag} check
```
If everything is ok, you will see that all required checks are passed
and your environment is ready to run the oracle.
Expand Down Expand Up @@ -142,25 +142,28 @@ Full variables list could be found [here](https://github.com/lidofinance/lido-or

## Env variables

| Name | Description | Required | Example value |
|----------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------------------------|
| `EXECUTION_CLIENT_URI` | URI of the Execution Layer client | True | `http://localhost:8545` |
| `CONSENSUS_CLIENT_URI` | URI of the Consensus Layer client | True | `http://localhost:5052` |
| `KEYS_API_URI` | URI of the Keys API | True | `http://localhost:8080` |
| `LIDO_LOCATOR_ADDRESS` | Address of the Lido contract | True | `0x1...` |
| `MEMBER_PRIV_KEY` | Private key of the Oracle member account | False | `0x1...` |
| `MEMBER_PRIV_KEY_FILE` | A path to the file contained the private key of the Oracle member account. It takes precedence over `MEMBER_PRIV_KEY` | False | `/app/private_key` |
| `FINALIZATION_BATCH_MAX_REQUEST_COUNT` | The size of the batch to be finalized per request (The larger the batch size, the more memory of the contract is used but the fewer requests are needed) | False | `1000` |
| `ALLOW_REPORTING_IN_BUNKER_MODE` | Allow the Oracle to do report if bunker mode is active | False | `True` |
| `TX_GAS_ADDITION` | Used to modify gas parameter that used in transaction. (gas = estimated_gas + TX_GAS_ADDITION) | False | `1.75` |
| `CYCLE_SLEEP_IN_SECONDS` | The time between cycles of the oracle's activity | False | `12` |
| `SUBMIT_DATA_DELAY_IN_SLOTS` | The difference in slots between submit data transactions from Oracles. It is used to prevent simultaneous sending of transactions and, as a result, transactions revert. | False | `6` |
| `HTTP_REQUEST_RETRY_COUNT` | Total number of retries to fetch data from endpoint | False | `5` |
| `HTTP_REQUEST_SLEEP_BEFORE_RETRY_IN_SECONDS` | The delay http provider sleeps if API is stuck | False | `12` |
| `HTTP_REQUEST_TIMEOUT` | Timeout for HTTP requests | False | `300` |
| `PRIORITY_FEE_PERCENTILE` | Priority fee percentile from prev block that would be used to send tx | False | `3` |
| `MIN_PRIORITY_FEE` | Min priority fee that would be used to send tx | False | `50000000` |
| `MAX_PRIORITY_FEE` | Max priority fee that would be used to send tx | False | `100000000000` |
| Name | Description | Required | Example value |
|--------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------------------------|
| `EXECUTION_CLIENT_URI` | URI of the Execution Layer client | True | `http://localhost:8545` |
| `CONSENSUS_CLIENT_URI` | URI of the Consensus Layer client | True | `http://localhost:5052` |
| `KEYS_API_URI` | URI of the Keys API | True | `http://localhost:8080` |
| `LIDO_LOCATOR_ADDRESS` | Address of the Lido contract | True | `0x1...` |
| `MEMBER_PRIV_KEY` | Private key of the Oracle member account | False | `0x1...` |
| `MEMBER_PRIV_KEY_FILE` | A path to the file contained the private key of the Oracle member account. It takes precedence over `MEMBER_PRIV_KEY` | False | `/app/private_key` |
| `FINALIZATION_BATCH_MAX_REQUEST_COUNT` | The size of the batch to be finalized per request (The larger the batch size, the more memory of the contract is used but the fewer requests are needed) | False | `1000` |
| `ALLOW_REPORTING_IN_BUNKER_MODE` | Allow the Oracle to do report if bunker mode is active | False | `True` |
| `TX_GAS_ADDITION` | Used to modify gas parameter that used in transaction. (gas = estimated_gas + TX_GAS_ADDITION) | False | `100000` |
| `CYCLE_SLEEP_IN_SECONDS` | The time between cycles of the oracle's activity | False | `12` |
| `SUBMIT_DATA_DELAY_IN_SLOTS` | The difference in slots between submit data transactions from Oracles. It is used to prevent simultaneous sending of transactions and, as a result, transactions revert. | False | `6` |
| `HTTP_REQUEST_TIMEOUT_CONSENSUS` | Timeout for HTTP consensus layer requests | False | `300` |
| `HTTP_REQUEST_RETRY_COUNT_CONSENSUS` | Total number of retries to fetch data from endpoint for consensus layer requests | False | `5` |
| `HTTP_REQUEST_SLEEP_BEFORE_RETRY_IN_SECONDS_CONSENSUS` | The delay http provider sleeps if API is stuck for consensus layer | False | `12` |
| `HTTP_REQUEST_TIMEOUT_KEYS_API` | Timeout for HTTP keys api requests | False | `10` |
| `HTTP_REQUEST_RETRY_COUNT_KEYS_API` | Total number of retries to fetch data from endpoint for keys api requests | False | `300` |
| `HTTP_REQUEST_SLEEP_BEFORE_RETRY_IN_SECONDS_KEYS_API` | The delay http provider sleeps if API is stuck for keys api | False | `300` |
| `PRIORITY_FEE_PERCENTILE` | Priority fee percentile from prev block that would be used to send tx | False | `3` |
| `MIN_PRIORITY_FEE` | Min priority fee that would be used to send tx | False | `50000000` |
| `MAX_PRIORITY_FEE` | Max priority fee that would be used to send tx | False | `100000000000` |
### Alerts
Expand All @@ -178,7 +181,7 @@ groups:
severity: critical
annotations:
summary: "Dangerously low account balance"
description: "Account balance is less than 3 ETH. Address: {.labels.address}: {.value} ETH"
description: "Account balance is less than 1 ETH. Address: {.labels.address}: {.value} ETH"
- alert: OutdatedData
expr: (lido_oracle_genesis_time + ignoring (state) lido_oracle_slot_number{state="head"} * 12) < time() - 300
for: 1h
Expand Down
32 changes: 16 additions & 16 deletions src/main.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
from typing import cast

from prometheus_client import start_http_server
from web3_multi_provider import FallbackProvider
from web3.middleware import simple_cache_middleware

from src import variables
Expand All @@ -20,6 +19,7 @@
ConsensusClientModule,
KeysAPIClientModule,
LidoValidatorsProvider,
FallbackProviderModule
)
from src.web3py.middleware import metrics_collector
from src.web3py.typings import Web3
Expand Down Expand Up @@ -57,7 +57,7 @@ def main(module: OracleModule):
start_http_server(variables.PROMETHEUS_PORT)

logger.info({'msg': 'Initialize multi web3 provider.'})
web3 = Web3(FallbackProvider(variables.EXECUTION_CLIENT_URI))
web3 = Web3(FallbackProviderModule(variables.EXECUTION_CLIENT_URI))

logger.info({'msg': 'Modify web3 with custom contract function call.'})
tweak_w3_contracts(web3)
Expand All @@ -68,6 +68,8 @@ def main(module: OracleModule):
logger.info({'msg': 'Initialize keys api client.'})
kac = KeysAPIClientModule(variables.KEYS_API_URI, web3)

check_providers_chain_ids(web3, cc, kac)

web3.attach_modules({
'lido_contracts': LidoContracts,
'lido_validators': LidoValidatorsProvider,
Expand All @@ -81,7 +83,6 @@ def main(module: OracleModule):
web3.middleware_onion.add(simple_cache_middleware)

logger.info({'msg': 'Sanity checks.'})
check_providers_chain_ids(web3)

if module == OracleModule.ACCOUNTING:
logger.info({'msg': 'Initialize Accounting module.'})
Expand All @@ -101,19 +102,18 @@ def check():
return ChecksModule().execute_module()


def check_providers_chain_ids(web3: Web3):
execution_chain_id = web3.eth.chain_id
consensus_chain_id = int(web3.cc.get_config_spec().DEPOSIT_CHAIN_ID)
chain_ids = [
Web3.to_int(hexstr=provider.make_request("eth_chainId", []).get('result'))
for provider in cast(FallbackProvider, web3.provider)._providers # type: ignore[attr-defined] # pylint: disable=protected-access
]
keys_api_chain_id = web3.kac.get_status().chainId
if any(execution_chain_id != chain_id for chain_id in [*chain_ids, consensus_chain_id, keys_api_chain_id]):
raise ValueError('Different chain ids detected:\n'
f'Execution chain ids: {", ".join(map(str, chain_ids))}\n'
f'Consensus chain id: {consensus_chain_id}\n'
f'Keys API chain id: {keys_api_chain_id}\n')
def check_providers_chain_ids(web3: Web3, cc: ConsensusClientModule, kac: KeysAPIClientModule):
keys_api_chain_id = kac.check_providers_consistency()
consensus_chain_id = cc.check_providers_consistency()
execution_chain_id = cast(FallbackProviderModule, web3.provider).check_providers_consistency()

if execution_chain_id == consensus_chain_id == keys_api_chain_id:
return

raise ValueError('Different chain ids detected:\n'
f'Execution chain id: {execution_chain_id}\n'
f'Consensus chain id: {consensus_chain_id}\n'
f'Keys API chain id: {keys_api_chain_id}\n')


if __name__ == '__main__':
Expand Down
9 changes: 7 additions & 2 deletions src/modules/accounting/accounting.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -71,10 +71,15 @@ def execute_module(self, last_finalized_blockstamp: BlockStamp) -> ModuleExecute
return ModuleExecuteDelay.NEXT_FINALIZED_EPOCH

def process_extra_data(self, blockstamp: ReferenceBlockStamp):
latest_blockstamp = self._get_latest_blockstamp()
if not self.can_submit_extra_data(latest_blockstamp):
logger.info({'msg': 'Extra data can not be submitted.'})
return

chain_config = self.get_chain_config(blockstamp)
slots_to_sleep = self._get_slot_delay_before_data_submit(blockstamp)
seconds_to_sleep = slots_to_sleep * chain_config.seconds_per_slot
logger.info({'msg': f'Sleep for {seconds_to_sleep} before sending extra data.'})
logger.info({'msg': f'Sleep for {seconds_to_sleep} seconds before sending extra data.'})
sleep(seconds_to_sleep)

latest_blockstamp = self._get_latest_blockstamp()
Expand Down Expand Up @@ -237,7 +242,7 @@ def simulate_cl_rebase(self, blockstamp: ReferenceBlockStamp) -> LidoReportRebas
Simulate rebase excluding any execution rewards.
This used to check worst scenarios in bunker service.
"""
return self.simulate_rebase_after_report(blockstamp, Wei(0))
return self.simulate_rebase_after_report(blockstamp, el_rewards=Wei(0))

def simulate_full_rebase(self, blockstamp: ReferenceBlockStamp) -> LidoReportRebase:
el_rewards = self.w3.lido_contracts.get_el_vault_balance(blockstamp)
Expand Down
2 changes: 1 addition & 1 deletion src/modules/checks/suites/common.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ def ejector(web3, skip_locator):

def check_providers_chain_ids(web3):
"""Make sure all providers are on the same chain"""
chain_ids_check(web3)
chain_ids_check(web3, web3.cc, web3.kac)


def check_accounting_contract_configs(accounting):
Expand Down
6 changes: 6 additions & 0 deletions src/providers/consensus/client.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -156,3 +156,9 @@ def __raise_last_missed_slot_error(self, errors: list[Exception]) -> Exception |
return error

return None

def _get_chain_id_with_provider(self, provider_index: int) -> int:
data, _ = self._get_without_fallbacks(self.hosts[provider_index], self.API_GET_SPEC)
if not isinstance(data, dict):
raise ValueError("Expected mapping response from getSpec")
return int(BeaconSpecResponse.from_response(**data).DEPOSIT_CHAIN_ID)
29 changes: 23 additions & 6 deletions src/providers/http_provider.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@
from requests.exceptions import ConnectionError as RequestsConnectionError
from urllib3 import Retry

from src.variables import HTTP_REQUEST_RETRY_COUNT, HTTP_REQUEST_SLEEP_BEFORE_RETRY_IN_SECONDS, HTTP_REQUEST_TIMEOUT
from src.web3py.extensions.consistency import ProviderConsistencyModule


logger = logging.getLogger(__name__)

Expand All @@ -29,22 +30,32 @@ def __init__(self, *args, status: int, text: str):
super().__init__(*args)


class HTTPProvider(ABC):
class HTTPProvider(ProviderConsistencyModule, ABC):
"""
Base HTTP Provider with metrics and retry strategy integrated inside.
"""
PROMETHEUS_HISTOGRAM: Histogram
request_timeout: int

def __init__(self, hosts: list[str]):
def __init__(
self,
hosts: list[str],
request_timeout: int,
retry_total: int,
retry_backoff_factor: int,
):
if not hosts:
raise NoHostsProvided(f"No hosts provided for {self.__class__.__name__}")

self.hosts = hosts
self.request_timeout = request_timeout
self.retry_count = retry_total
self.backoff_factor = retry_backoff_factor

retry_strategy = Retry(
total=HTTP_REQUEST_RETRY_COUNT,
total=self.retry_count,
status_forcelist=[418, 429, 500, 502, 503, 504],
backoff_factor=HTTP_REQUEST_SLEEP_BEFORE_RETRY_IN_SECONDS,
backoff_factor=self.backoff_factor,
)

adapter = HTTPAdapter(max_retries=retry_strategy)
Expand Down Expand Up @@ -113,7 +124,7 @@ def _get_without_fallbacks(
response = self.session.get(
self._urljoin(host, complete_endpoint if path_params else endpoint),
params=query_params,
timeout=HTTP_REQUEST_TIMEOUT,
timeout=self.request_timeout,
)
except RequestsConnectionError as error:
logger.debug({'msg': str(error)})
Expand Down Expand Up @@ -151,3 +162,9 @@ def _get_without_fallbacks(
meta = {}

return data, meta

def get_all_providers(self) -> list[str]:
return self.hosts

def _get_chain_id_with_provider(self, provider_index: int) -> int:
raise NotImplementedError("_chain_id should be implemented")
Loading

0 comments on commit 0661a89

Please sign in to comment.