Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat/ds scoring #218

Merged
merged 102 commits into from
Apr 1, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
102 commits
Select commit Hold shift + click to select a range
a75036f
feat: initial commit
butonium Mar 15, 2024
85fd8f5
fix
butonium Mar 15, 2024
a6f5333
fix
butonium Mar 15, 2024
e238aff
fix
butonium Mar 15, 2024
5ca23f3
fix
butonium Mar 15, 2024
fecd9bb
Update v2-prepare-scoring.yml
butonium Mar 15, 2024
6254167
temp trigger
butonium Mar 15, 2024
258607e
fix
butonium Mar 15, 2024
26761b8
revert
butonium Mar 15, 2024
b577964
change trigger
butonium Mar 15, 2024
1aea567
remove user tag
butonium Mar 15, 2024
ea5d80f
add create pr
butonium Mar 15, 2024
7c4c9fa
change target branch to scoring-v2
butonium Mar 15, 2024
1f39650
add checkout scoring-v2
butonium Mar 15, 2024
8742586
oops
butonium Mar 15, 2024
62640ac
add gzip
butonium Mar 15, 2024
bfa36ff
change scoring branch
butonium Mar 15, 2024
2529976
add token
butonium Mar 15, 2024
34a5670
debug
butonium Mar 15, 2024
6f85f53
fix
butonium Mar 15, 2024
94a3a31
fix
butonium Mar 15, 2024
41911bc
fix
butonium Mar 15, 2024
30b3a41
Merge branch 'feat/ds-scoring' into scoring-v2
butonium Mar 15, 2024
10422a6
fix pr naming + add discord
butonium Mar 15, 2024
2a1c2ea
fix
butonium Mar 15, 2024
aa0df0f
fix
butonium Mar 15, 2024
87d09a8
test apply scoring v2
butonium Mar 15, 2024
c2ff4da
fix
butonium Mar 15, 2024
b390548
Publish V2 Scoring Results (588.292421) (#220)
github-actions[bot] Mar 15, 2024
74983b4
fix branch name
butonium Mar 15, 2024
274111c
fix: branch name
butonium Mar 15, 2024
147508d
fix
butonium Mar 15, 2024
06e6e15
Publish V2 Scoring Results (588.309697) (#222)
github-actions[bot] Mar 15, 2024
b422861
check if params exists
butonium Mar 15, 2024
9ac6fbd
Merge branch 'feat/ds-scoring' into scoring-v2
butonium Mar 15, 2024
b96dec4
Publish V2 Scoring Results (588.311033) (#223)
github-actions[bot] Mar 15, 2024
06cf670
temp disable notifications
butonium Mar 15, 2024
e378da3
Publish V2 Scoring Results (588.313063) (#224)
github-actions[bot] Mar 15, 2024
4da25b8
fix
butonium Mar 15, 2024
1523b8c
Merge branch 'scoring-v2' into feat/ds-scoring
butonium Mar 15, 2024
5b17dd8
Publish V2 Scoring Results (588.314516) (#225)
github-actions[bot] Mar 15, 2024
bbc26ac
fix
butonium Mar 15, 2024
6971133
Publish V2 Scoring Results (588.315786) (#226)
github-actions[bot] Mar 15, 2024
bf44e5d
fix
butonium Mar 15, 2024
153cdbc
Publish V2 Scoring Results (588.316540) (#227)
github-actions[bot] Mar 15, 2024
1cd04c4
add simulation
butonium Mar 15, 2024
059d3d6
Publish V2 Scoring Results (588.317500) (#228)
github-actions[bot] Mar 15, 2024
5177aae
testing components
butonium Mar 15, 2024
1e10c30
Publish V2 Scoring Results (588.318434) (#229)
github-actions[bot] Mar 15, 2024
c64fdca
test discord notification
butonium Mar 15, 2024
5c8a6bf
Publish V2 Scoring Results (588.323852) (#230)
github-actions[bot] Mar 15, 2024
5837c8f
fix env
butonium Mar 15, 2024
9b2f00d
Publish V2 Scoring Results (588.325268) (#231)
github-actions[bot] Mar 15, 2024
ba26bd6
fix missing branch
butonium Mar 15, 2024
257a804
Publish V2 Scoring Results (588.326175) (#232)
github-actions[bot] Mar 15, 2024
4d4af6f
possible fix for last error
butonium Mar 15, 2024
486825e
Publish V2 Scoring Results (588.328517) (#233)
github-actions[bot] Mar 15, 2024
18c371c
fix
butonium Mar 15, 2024
8064bb8
Publish V2 Scoring Results (588.329744) (#234)
github-actions[bot] Mar 15, 2024
ccfbc41
debug
butonium Mar 15, 2024
7dbae5e
Publish V2 Scoring Results (588.333694) (#235)
github-actions[bot] Mar 15, 2024
7246003
test decoupling
butonium Mar 27, 2024
8b687a6
Publish V2 Scoring Results (594.149675) (#241)
github-actions[bot] Mar 27, 2024
c2a0681
temp remove notifs
butonium Mar 27, 2024
0a4f7b4
Publish V2 Scoring Results (594.153355) (#242)
github-actions[bot] Mar 27, 2024
ae45288
debug
butonium Mar 27, 2024
1edbeb5
Publish V2 Scoring Results (594.154504) (#243)
github-actions[bot] Mar 27, 2024
2ef7c4c
debug
butonium Mar 27, 2024
336e871
Publish V2 Scoring Results (594.155541) (#244)
github-actions[bot] Mar 27, 2024
5d8e25f
fix merging and auth
butonium Mar 27, 2024
df4d3ca
Publish V2 Scoring Results (594.157826) (#245)
github-actions[bot] Mar 27, 2024
c7604a1
fix apply scores
butonium Mar 28, 2024
701e24c
Publish V2 Scoring Results (594.323041) (#247)
github-actions[bot] Mar 28, 2024
8b28601
fix: apply score request
butonium Mar 28, 2024
101f34d
Publish V2 Scoring Results (594.334130) (#249)
github-actions[bot] Mar 28, 2024
61d591f
fix: scoring_id
butonium Mar 28, 2024
32823c3
Publish V2 Scoring Results (594.336539) (#250)
github-actions[bot] Mar 28, 2024
7d7a084
fix: unstakes pr creation
butonium Mar 28, 2024
115ed51
Publish V2 Scoring Results (594.340757) (#251)
github-actions[bot] Mar 28, 2024
1e4ead7
fix: merging
butonium Mar 28, 2024
b1b969f
Publish V2 Scoring Results (594.342993) (#252)
github-actions[bot] Mar 28, 2024
d50f314
fix: unstakes summary
butonium Mar 28, 2024
4f51e1f
Publish V2 Scoring Results (594.346527) (#254)
github-actions[bot] Mar 28, 2024
5ced0b6
Merge branch 'master' into feat/ds-scoring
butonium Mar 28, 2024
c2c2d85
Merge branch 'scoring-v2' into feat/ds-scoring
butonium Mar 28, 2024
ba48c32
remove unused files
butonium Mar 28, 2024
c965b55
remove old scoring files
butonium Mar 28, 2024
faf3b92
remove unused files
butonium Mar 28, 2024
be7fe93
Merge branch 'scoring-v2' into feat/ds-scoring
butonium Mar 28, 2024
27e968f
add apply late unstakes
butonium Mar 28, 2024
f0bc643
Publish V2 Scoring Results (594.363701) (#258)
github-actions[bot] Mar 28, 2024
6ceb8ec
temp fix
butonium Mar 28, 2024
6b48e20
Publish V2 Scoring Results (594.366768) (#259)
github-actions[bot] Mar 28, 2024
677165c
emergency unstakes for scoring run 594.366768 (#260)
github-actions[bot] Mar 28, 2024
99f29b3
fix: change triggers
butonium Mar 28, 2024
d3facc1
uncomment
butonium Mar 28, 2024
c147b7b
Merge branch 'master' into scoring-v2
butonium Mar 28, 2024
19c263c
Merge branch 'scoring-v2' into feat/ds-scoring
butonium Mar 28, 2024
d82b9ae
remove scoring runs
butonium Mar 28, 2024
da2da3f
Merge branch 'scoring-v2' into feat/ds-scoring
butonium Mar 28, 2024
ae9dab3
missing blacklists
butonium Mar 28, 2024
5c528d9
fix: apply review
butonium Apr 1, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
221 changes: 221 additions & 0 deletions .github/workflows/v2-apply-scoring.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,221 @@
name: Apply Scoring V2

on:
workflow_dispatch:
pull_request:
types: [closed]
branches:
- scoring-v2

jobs:
apply_scoring_v2:
runs-on: ubuntu-latest
if: github.event_name == 'workflow_dispatch' || (github.event_name == 'pull_request' && github.event.pull_request.merged == true && startsWith(github.event.pull_request.head.ref, 'ds-scoring/'))

steps:
- name: Checkout the repository
uses: actions/checkout@master
with:
ref: scoring-v2
fetch-depth: 20

- name: Load scoring details
run: |
set -ex
git fetch --all
scores_csv=$(git log -1 origin/scoring-v2 --name-only --pretty=format: --grep 'scoring run' | grep scores.csv)
params_json=$(git log -1 origin/scoring-v2 --name-only --pretty=format: --grep 'scoring run' | grep params.json)

scoring_run_ui_id=$(git log -1 origin/scoring-v2 --name-only --pretty=format: --grep 'scoring run' | head -6 | tail -1 | awk -F / '{print $2}')
epoch=$(<<<"$scoring_run_ui_id" awk -F . '{print $1}')

weightVoteCredits=$(jq '.weightVoteCredits' $params_json)
weightInflationCommission=$(jq '.weightInflationCommission' $params_json)
weightMEVCommission=$(jq '.weightMEVCommission' $params_json)
weightStakeConcentrationCity=$(jq '.weightStakeConcentrationCity' $params_json)
weightStakeConcentrationASO=$(jq '.weightStakeConcentrationASO' $params_json)
weightStakeConcentrationNode=$(jq '.weightStakeConcentrationNode' $params_json)
weightStakeConcentrationCountry=$(jq '.weightStakeConcentrationCountry' $params_json)
weightBlockProduction=$(jq '.weightBlockProduction' $params_json)

if [[ -z $scores_csv ]]
then
echo "Failed to find CSV with scores in the PR!"
exit 1
fi

if [[ -z $scoring_run_ui_id ]]
then
echo "Failed to detect scoring run UI ID from the PR!"
exit 1
fi

if [[ -z $epoch ]]
then
echo "Failed to detect epoch from the UI ID!"
exit 1
fi

echo "scores_csv=$scores_csv" >> $GITHUB_ENV
echo "scoring_run_ui_id=$scoring_run_ui_id" >> $GITHUB_ENV
echo "epoch=$epoch" >> $GITHUB_ENV
echo "components=COMMISSION_ADJUSTED_CREDITS,BLOCK_PRODUCTION,INFLATION_COMMISSION,MEV_COMMISSION,DC_COUNTRY_CONCENTRATION,DC_CITY_CONCENTRATION,DC_ASO_CONCENTRATION,DC_NODE_CONCENTRATION" >> $GITHUB_ENV
echo "components_weights=$weightVoteCredits,$weightBlockProduction,$weightInflationCommission,$weightMEVCommission,$weightStakeConcentrationCountry,$weightStakeConcentrationCity,$weightStakeConcentrationASO,$weightStakeConcentrationNode" >> $GITHUB_ENV

- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_DEFAULT_REGION }}

- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v1

- name: Prepare solana config
run: |
cat <<EOF > /tmp/solana-config.yml
json_rpc_url: "$RPC_URL"
websocket_url: ""
keypair_path: /.config/solana/id.json
address_labels:
"11111111111111111111111111111111": System Program
commitment: confirmed
EOF
echo "$KEYPAIR" > /tmp/id.json

env:
RPC_URL: ${{ secrets.RPC_URL }}
KEYPAIR: ${{ secrets.VALIDATOR_MANAGEMENT_KEYPAIR }}

- name: Configure image
run: |
images=$(aws ecr describe-images --repository-name marinade.finance/validator-manager)
latest=$(<<<"$images" jq '.imageDetails[] | .imagePushedAt + " " + .imageTags[0]' -r | sort | tail -1 | cut -d' ' -f2)
echo "latest=$latest" >> $GITHUB_ENV

- name: Run scoring - simulation
run: |
docker run --rm --user "$(id -u):$(id -g)" \
-v /tmp/solana-config.yml:/.config/solana/cli/config.yml \
-v /tmp/id.json:/.config/solana/id.json \
-v "$(realpath "$SCORES_CSV"):/scores.csv" \
"$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG" \
./validator-manager -s --print-only update-scores2 --scores-file /scores.csv
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
ECR_REPOSITORY: marinade.finance/validator-manager
SCORES_CSV: ${{ env.scores_csv }}
IMAGE_TAG: ${{ env.latest }}

- name: Publish scoring results
run: |
curl -sLfS "https://scoring-dev.marinade.finance/api/v1/scores/upload?epoch=${{ env.epoch }}&components=${{ env.components }}&component_weights=${{ env.components_weights }}&ui_id=${{ env.scoring_run_ui_id }}" -X POST \
-H "Authorization: Bearer ${{ secrets.VALIDATORS_API_ADMIN_TOKEN }}" \
-F "scores=@${{ env.scores_csv }}"

early_unstake:
needs: apply_scoring_v2
runs-on: ubuntu-latest
steps:
- name: Checkout the repository
uses: actions/checkout@master
with:
ref: scoring-v2
fetch-depth: 20

- name: Load scoring details
run: |
set -ex
unstake_hints_json=$(git log -1 --name-only --pretty=format: --grep 'scoring run' | grep unstake-hints.json)
scoring_run_ui_id=$(git log -1 origin/scoring-v2 --name-only --pretty=format: --grep 'scoring run' | head -6 | tail -1 | awk -F / '{print $2}')
epoch=$(<<<"$scoring_run_ui_id" awk -F . '{print $1}')

if [[ -z $unstake_hints_json ]]
then
echo "Failed to find JSON with unstake hints in the PR!"
exit 1
fi

if [[ -z $epoch ]]
then
echo "Failed to detect epoch from the UI ID!"
exit 1
fi

echo "unstake_hints_json=$unstake_hints_json" >> $GITHUB_ENV
echo "epoch=$epoch" >> $GITHUB_ENV

- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_DEFAULT_REGION }}

- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v1

- name: Prepare solana config
run: |
cat <<EOF > /tmp/solana-config.yml
json_rpc_url: "$RPC_URL"
websocket_url: ""
keypair_path: /.config/solana/id.json
address_labels:
"11111111111111111111111111111111": System Program
commitment: confirmed
EOF
echo "$KEYPAIR" > /tmp/id.json

env:
RPC_URL: ${{ secrets.RPC_URL }}
KEYPAIR: ${{ secrets.VALIDATOR_MANAGEMENT_KEYPAIR }}

- name: Configure image
run: |
images=$(aws ecr describe-images --repository-name marinade.finance/validator-manager)
latest=$(<<<"$images" jq '.imageDetails[] | .imagePushedAt + " " + .imageTags[0]' -r | sort | tail -1 | cut -d' ' -f2)
echo "latest=$latest" >> $GITHUB_ENV

- name: Early unstake V2 - simulation
run: |
<"$UNSTAKE_HINTS_JSON" jq '.unstake_hints[].vote_account' -r | xargs -I{} \
docker run --rm --user "$(id -u):$(id -g)" \
-v /tmp/solana-config.yml:/.config/solana/cli/config.yml \
-v /tmp/id.json:/.config/solana/id.json \
"$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG" \
./validator-manager -s --print-only emergency-unstake {}
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
ECR_REPOSITORY: marinade.finance/validator-manager
UNSTAKE_HINTS_JSON: ${{ env.unstake_hints_json }}
IMAGE_TAG: ${{ env.latest }}

- name: Send Discord Notification
run: |
validators_count=$(<"$UNSTAKE_HINTS_JSON" jq '[.unstake_hints[].marinade_stake] | length' -r)
unstaked_sol=$(<"$UNSTAKE_HINTS_JSON" jq '[.unstake_hints[].marinade_stake] | add' -r)

message="Early unstake successfully applied. ($validators_count validators, $unstaked_sol SOL)"

if (( $validators_count == 0 ))
then
message="No emergency unstakes needed this time."
fi

curl "$DISCORD_WEBHOOK" -H "Content-Type: application/json" -d '{
"username": "Delegation Strategy - Early Unstake V2",
"avatar_url": "https://public.marinade.finance/ds-emergency-bot.png",
"embeds": [
{
"title": "'"$message"'",
"color": "52224"
}
]
}'
env:
DISCORD_WEBHOOK: ${{ secrets.DISCORD_WEBHOOK }}
UNSTAKE_HINTS_JSON: ${{ env.unstake_hints_json }}
115 changes: 115 additions & 0 deletions .github/workflows/v2-apply-unstakes.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@

name: Apply Unstakes V2

on:
workflow_dispatch:
pull_request:
types: [closed]
branches:
- scoring-v2

jobs:
late_unstake:
if: startsWith(github.event.pull_request.head.ref, 'ds-unstakes') && github.event.pull_request.merged == true
runs-on: ubuntu-latest
steps:
- name: Checkout the repository
uses: actions/checkout@master
with:
ref: scoring-v2
fetch-depth: 20

- name: Load scoring details
run: |
set -ex
unstake_hints_json=$(git log -1 --name-only --pretty=format: --grep 'emergency unstakes' | grep unstake-hints.json)
scoring_run_ui_id=$(git log -1 origin/scoring-v2 --name-only --pretty=format: --grep 'emergency unstakes' | head -3 | tail -1 | awk -F / '{print $2}')
epoch=$(<<<"$scoring_run_ui_id" awk -F . '{print $1}')

if [[ -z $unstake_hints_json ]]
then
echo "Failed to find JSON with unstake hints in the PR!"
exit 1
fi

if [[ -z $epoch ]]
then
echo "Failed to detect epoch from the UI ID!"
exit 1
fi

echo "unstake_hints_json=$unstake_hints_json" >> $GITHUB_ENV
echo "epoch=$epoch" >> $GITHUB_ENV

- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_DEFAULT_REGION }}

- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v1

- name: Prepare solana config
run: |
cat <<EOF > /tmp/solana-config.yml
json_rpc_url: "$RPC_URL"
websocket_url: ""
keypair_path: /.config/solana/id.json
address_labels:
"11111111111111111111111111111111": System Program
commitment: confirmed
EOF
echo "$KEYPAIR" > /tmp/id.json

env:
RPC_URL: ${{ secrets.RPC_URL }}
KEYPAIR: ${{ secrets.VALIDATOR_MANAGEMENT_KEYPAIR }}

- name: Configure image
run: |
images=$(aws ecr describe-images --repository-name marinade.finance/validator-manager)
latest=$(<<<"$images" jq '.imageDetails[] | .imagePushedAt + " " + .imageTags[0]' -r | sort | tail -1 | cut -d' ' -f2)
echo "latest=$latest" >> $GITHUB_ENV

- name: Early unstake V2 - simulation
run: |
<"$UNSTAKE_HINTS_JSON" jq '.unstake_hints[].vote_account' -r | xargs -I{} \
docker run --rm --user "$(id -u):$(id -g)" \
-v /tmp/solana-config.yml:/.config/solana/cli/config.yml \
-v /tmp/id.json:/.config/solana/id.json \
"$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG" \
./validator-manager -s --print-only emergency-unstake {}
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
ECR_REPOSITORY: marinade.finance/validator-manager
UNSTAKE_HINTS_JSON: ${{ env.unstake_hints_json }}
IMAGE_TAG: ${{ env.latest }}

- name: Send Discord Notification
run: |
validators_count=$(<"$UNSTAKE_HINTS_JSON" jq '[.unstake_hints[].marinade_stake] | length' -r)
unstaked_sol=$(<"$UNSTAKE_HINTS_JSON" jq '[.unstake_hints[].marinade_stake] | add' -r)

message="Late unstake successfully applied. ($validators_count validators, $unstaked_sol SOL)"

if (( $validators_count == 0 ))
then
message="No emergency unstakes needed this time."
fi

curl "$DISCORD_WEBHOOK" -H "Content-Type: application/json" -d '{
"username": "Delegation Strategy - Early Unstake V2",
"avatar_url": "https://public.marinade.finance/ds-emergency-bot.png",
"embeds": [
{
"title": "'"$message"'",
"color": "52224"
}
]
}'
env:
DISCORD_WEBHOOK: ${{ secrets.DISCORD_WEBHOOK }}
UNSTAKE_HINTS_JSON: ${{ env.unstake_hints_json }}
Loading