-
-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[DRAFT]: Make auto updates rate-limit aware #22994
base: master
Are you sure you want to change the base?
Conversation
|
I know we don't want to spam logs, I'm not suggesting this is ready to merge in this state, this is an initial draft. If you got any code suggestions do feel free to put them in here, or push them to my branch, or I can move the branch over to this repository if that is easier in terms of workflow. |
I can try this a bit later, currently I should go to sleep. |
Yeah I'm about to as well, I just wanted to push this before I do. |
Alright I did a full run now that the limit reset for me and it looks like it takes ~820 API calls for the full auto-update cycle. |
That limit is shared globally between all CI runs, and as such also resets globally for all CI runs. (That limit being 5000 per hour) |
@@ -183,12 +186,12 @@ _fetch_and_cache_tags() { | |||
echo "build run_all: phony ${__PACKAGES[@]}" | |||
echo "default run_all" | |||
} | |||
|
|||
local LATEST_TAGS="$(\ | |||
F="$(declare -p TERMUX_SCRIPTDIR GITHUB_TOKEN TERMUX_REPOLOGY_DATA_FILE); $(declare -f __main__ | sed 's/__main__ ()//')" \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the main reason we are getting rate limited is because we aren't slowing down our api requests. See https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2022-11-28#about-secondary-rate-limits, maybe we should actually fix this than using such workarounds
f9cc350
to
bfe5130
Compare
I changed it now so it only outputs the rate limit information when the package has a GitHub source URL. |
scripts/bin/update-packages
Outdated
F="$(declare -p TERMUX_SCRIPTDIR GITHUB_TOKEN TERMUX_REPOLOGY_DATA_FILE); $(declare -f __main__ | sed 's/__main__ ()//')" \ | ||
env --chdir=/tmp ninja -f /dev/stdin <<< "$(__generate__)" |& grep "^PKG|" | ||
env --chdir=/tmp ninja -f /dev/stdin < <(__generate__ |& grep "^PKG|") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
grep
must be applied to ninja
, not to __generate__
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep noticed that already locally just haven't pushed it yet.
bfe5130
to
f84e3dc
Compare
This latest version is based on @twaik's suggestion to use the GraphQL API for querying github packages. The GraphQL queries need to be sent as a JSON string and as such are very particular about containing no tabs or newlines. It still needs some cleanup, but a batch size of 100 repos per request seems to work consistently. The queries are built dynamically by the batching loop. Inside this dropdown I have provided a prettified JSON object for a single batch and the extracted GraphQL query
// Begin test: Sat Feb 1 01:10:15 PM UTC 2025
// Batch: 0
{
"query": "
fragment _latest_release_tag on Repository {
releases(first: 1, orderBy: {field: CREATED_AT, direction: DESC}) {
nodes { tagName }
}
}
fragment _newest_tag on Repository {
refs( refPrefix: \"refs/tags/\" first: 1 orderBy: {field: TAG_COMMIT_DATE, direction: DESC}) {
nodes { name }
}
}
query {
_0: repository(owner: \"mevdschee\", name: \"2048.c\") { ..._latest_release_tag }
_1: repository(owner: \"radare\", name: \"acr\") { ..._latest_release_tag }
_2: repository(owner: \"Qucs\", name: \"ADMS\") { ..._latest_release_tag }
_3: repository(owner: \"mbrubeck\", name: \"agate\") { ..._latest_release_tag }
_4: repository(owner: \"FiloSottile\", name: \"age\") { ..._latest_release_tag }
_5: repository(owner: \"asciinema\", name: \"agg\") { ..._latest_release_tag }
_6: repository(owner: \"theZiz\", name: \"aha\") { ..._newest_tag }
_7: repository(owner: \"sigoden\", name: \"aichat\") { ..._latest_release_tag }
_8: repository(owner: \"alembic\", name: \"alembic\") { ..._latest_release_tag }
_9: repository(owner: \"xyproto\", name: \"algernon\") { ..._latest_release_tag }
_10: repository(owner: \"alsa-project\", name: \"alsa-lib\") { ..._newest_tag }
_11: repository(owner: \"alsa-project\", name: \"alsa-plugins\") { ..._newest_tag }
_12: repository(owner: \"alsa-project\", name: \"alsa-utils\") { ..._newest_tag }
_13: repository(owner: \"dalance\", name: \"amber\") { ..._latest_release_tag }
_14: repository(owner: \"makeworld-the-better-one\", name: \"amfora\") { ..._latest_release_tag }
_15: repository(owner: \"nmeum\", name: \"android-tools\") { ..._latest_release_tag }
_16: repository(owner: \"ysf\", name: \"anewer\") { ..._latest_release_tag }
_17: repository(owner: \"rcoh\", name: \"angle-grinder\") { ..._latest_release_tag }
_18: repository(owner: \"pystardust\", name: \"ani-cli\") { ..._latest_release_tag }
_19: repository(owner: \"getantibody\", name: \"antibody\") { ..._latest_release_tag }
_20: repository(owner: \"EFForg\", name: \"apkeep\") { ..._latest_release_tag }
_21: repository(owner: \"ximion\", name: \"appstream\") { ..._newest_tag }
_22: repository(owner: \"aptly-dev\", name: \"aptly\") { ..._latest_release_tag }
_23: repository(owner: \"argp-standalone\", name: \"argp-standalone\") { ..._newest_tag }
_24: repository(owner: \"aria2\", name: \"aria2\") { ..._newest_tag }
_25: repository(owner: \"opencollab\", name: \"arpack-ng\") { ..._newest_tag }
_26: repository(owner: \"asciidoc\", name: \"asciidoc-py3\") { ..._latest_release_tag }
_27: repository(owner: \"asciinema\", name: \"asciinema\") { ..._latest_release_tag }
_28: repository(owner: \"bergercookie\", name: \"asm-lsp\") { ..._newest_tag }
_29: repository(owner: \"assimp\", name: \"assimp\") { ..._latest_release_tag }
_30: repository(owner: \"OpenVisionE2\", name: \"astra-sm\") { ..._newest_tag }
_31: repository(owner: \"wez\", name: \"atomicparsley\") { ..._latest_release_tag }
_32: repository(owner: \"atomvm\", name: \"AtomVM\") { ..._latest_release_tag }
_33: repository(owner: \"ellie\", name: \"atuin\") { ..._latest_release_tag }
_34: repository(owner: \"wting\", name: \"autojump\") { ..._latest_release_tag }
_35: repository(owner: \"Ro5bert\", name: \"avra\") { ..._newest_tag }
_36: repository(owner: \"slavaGanzin\", name: \"await\") { ..._latest_release_tag }
_37: repository(owner: \"axel-download-accelerator\", name: \"axel\") { ..._latest_release_tag }
_38: repository(owner: \"BLAKE3-team\", name: \"BLAKE3\") { ..._latest_release_tag }
_39: repository(owner: \"fumiama\", name: \"base16384\") { ..._latest_release_tag }
_40: repository(owner: \"scop\", name: \"bash-completion\") { ..._latest_release_tag }
_41: repository(owner: \"sharkdp\", name: \"bat\") { ..._latest_release_tag }
_42: repository(owner: \"gavinhoward\", name: \"bc\") { ..._latest_release_tag }
_43: repository(owner: \"beanshell\", name: \"beanshell\") { ..._latest_release_tag }
_44: repository(owner: \"itchyny\", name: \"bed\") { ..._latest_release_tag }
_45: repository(owner: \"facebookincubator\", name: \"below\") { ..._newest_tag }
_46: repository(owner: \"rsharo\", name: \"bgrep\") { ..._latest_release_tag }
_47: repository(owner: \"louiz\", name: \"biboumi\") { ..._newest_tag }
_48: repository(owner: \"WebAssembly\", name: \"binaryen\") { ..._latest_release_tag }
_49: repository(owner: \"bitcoin\", name: \"bitcoin\") { ..._latest_release_tag }
_50: repository(owner: \"blade-lang\", name: \"blade\") { ..._latest_release_tag }
_51: repository(owner: \"jart\", name: \"blink\") { ..._latest_release_tag }
_52: repository(owner: \"blogc\", name: \"blogc\") { ..._latest_release_tag }
_53: repository(owner: \"tgraf\", name: \"bmon\") { ..._latest_release_tag }
_54: repository(owner: \"ekzhang\", name: \"bore\") { ..._latest_release_tag }
_55: repository(owner: \"borgbackup\", name: \"borg\") { ..._latest_release_tag }
_56: repository(owner: \"ascii-boxes\", name: \"boxes\") { ..._latest_release_tag }
_57: repository(owner: \"tmewett\", name: \"BrogueCE\") { ..._latest_release_tag }
_58: repository(owner: \"txthinking\", name: \"brook\") { ..._latest_release_tag }
_59: repository(owner: \"Canop\", name: \"broot\") { ..._latest_release_tag }
_60: repository(owner: \"google\", name: \"brotli\") { ..._latest_release_tag }
_61: repository(owner: \"bittorrent\", name: \"go-btfs\") { ..._latest_release_tag }
_62: repository(owner: \"bufbuild\", name: \"buf\") { ..._latest_release_tag }
_63: repository(owner: \"cabinpkg\", name: \"cabin\") { ..._latest_release_tag }
_64: repository(owner: \"caddyserver\", name: \"caddy\") { ..._latest_release_tag }
_65: repository(owner: \"lcn2\", name: \"calc\") { ..._latest_release_tag }
_66: repository(owner: \"c-ares\", name: \"c-ares\") { ..._latest_release_tag }
_67: repository(owner: \"karlstav\", name: \"cava\") { ..._latest_release_tag }
_68: repository(owner: \"kornelski\", name: \"cavif-rs\") { ..._latest_release_tag }
_69: repository(owner: \"ccache\", name: \"ccache\") { ..._latest_release_tag }
_70: repository(owner: \"0l1v3rr\", name: \"cli-file-manager\") { ..._latest_release_tag }
_71: repository(owner: \"CGAL\", name: \"cgal\") { ..._latest_release_tag }
_72: repository(owner: \"dloebl\", name: \"cgif\") { ..._latest_release_tag }
_73: repository(owner: \"hpjansson\", name: \"chafa\") { ..._latest_release_tag }
_74: repository(owner: \"libcheck\", name: \"check\") { ..._latest_release_tag }
_75: repository(owner: \"twpayne\", name: \"chezmoi\") { ..._latest_release_tag }
_76: repository(owner: \"theryangeary\", name: \"choose\") { ..._newest_tag }
_77: repository(owner: \"mitnk\", name: \"cicada\") { ..._latest_release_tag }
_78: repository(owner: \"CNugteren\", name: \"CLBlast\") { ..._latest_release_tag }
_79: repository(owner: \"leo-arch\", name: \"clifm\") { ..._latest_release_tag }
_80: repository(owner: \"Oblomov\", name: \"clinfo\") { ..._newest_tag }
_81: repository(owner: \"cloudflare\", name: \"cloudflared\") { ..._latest_release_tag }
_82: repository(owner: \"commonmark\", name: \"cmark\") { ..._latest_release_tag }
_83: repository(owner: \"abishekvashok\", name: \"cmatrix\") { ..._latest_release_tag }
_84: repository(owner: \"cmus\", name: \"cmus\") { ..._latest_release_tag }
_85: repository(owner: \"Arkq\", name: \"cmusfm\") { ..._newest_tag }
_86: repository(owner: \"exaexa\", name: \"codecrypt\") { ..._newest_tag }
_87: repository(owner: \"coin-or\", name: \"Cbc\") { ..._latest_release_tag }
_88: repository(owner: \"coin-or\", name: \"Clp\") { ..._latest_release_tag }
_89: repository(owner: \"cointop-sh\", name: \"cointop\") { ..._latest_release_tag }
_90: repository(owner: \"composer\", name: \"composer\") { ..._newest_tag }
_91: repository(owner: \"ros\", name: \"console_bridge\") { ..._latest_release_tag }
_92: repository(owner: \"cooklang\", name: \"cookcli\") { ..._latest_release_tag }
_93: repository(owner: \"DrakeW\", name: \"corgi\") { ..._latest_release_tag }
_94: repository(owner: \"cowsay-org\", name: \"cowsay\") { ..._latest_release_tag }
_95: repository(owner: \"danmar\", name: \"cppcheck\") { ..._latest_release_tag }
_96: repository(owner: \"Dr-Noob\", name: \"cpufetch\") { ..._latest_release_tag }
_97: repository(owner: \"opsengine\", name: \"cpulimit\") { ..._newest_tag }
_98: repository(owner: \"crawl\", name: \"crawl\") { ..._latest_release_tag }
_99: repository(owner: \"schollz\", name: \"croc\") { ..._latest_release_tag }
ratelimit: rateLimit { cost limit remaining used resetAt }
}"
} Extracted query: fragment _latest_release_tag on Repository {
releases(first: 1, orderBy: {field: CREATED_AT, direction: DESC}) {
nodes { tagName }
}
}
fragment _newest_tag on Repository {
refs( refPrefix: \"refs/tags/\" first: 1 orderBy: {field: TAG_COMMIT_DATE, direction: DESC}) {
nodes { name }
}
}
query {
_0: repository(owner: \"mevdschee\", name: \"2048.c\") { ..._latest_release_tag }
_1: repository(owner: \"radare\", name: \"acr\") { ..._latest_release_tag }
_2: repository(owner: \"Qucs\", name: \"ADMS\") { ..._latest_release_tag }
_3: repository(owner: \"mbrubeck\", name: \"agate\") { ..._latest_release_tag }
_4: repository(owner: \"FiloSottile\", name: \"age\") { ..._latest_release_tag }
_5: repository(owner: \"asciinema\", name: \"agg\") { ..._latest_release_tag }
_6: repository(owner: \"theZiz\", name: \"aha\") { ..._newest_tag }
_7: repository(owner: \"sigoden\", name: \"aichat\") { ..._latest_release_tag }
_8: repository(owner: \"alembic\", name: \"alembic\") { ..._latest_release_tag }
_9: repository(owner: \"xyproto\", name: \"algernon\") { ..._latest_release_tag }
_10: repository(owner: \"alsa-project\", name: \"alsa-lib\") { ..._newest_tag }
_11: repository(owner: \"alsa-project\", name: \"alsa-plugins\") { ..._newest_tag }
_12: repository(owner: \"alsa-project\", name: \"alsa-utils\") { ..._newest_tag }
_13: repository(owner: \"dalance\", name: \"amber\") { ..._latest_release_tag }
_14: repository(owner: \"makeworld-the-better-one\", name: \"amfora\") { ..._latest_release_tag }
_15: repository(owner: \"nmeum\", name: \"android-tools\") { ..._latest_release_tag }
_16: repository(owner: \"ysf\", name: \"anewer\") { ..._latest_release_tag }
_17: repository(owner: \"rcoh\", name: \"angle-grinder\") { ..._latest_release_tag }
_18: repository(owner: \"pystardust\", name: \"ani-cli\") { ..._latest_release_tag }
_19: repository(owner: \"getantibody\", name: \"antibody\") { ..._latest_release_tag }
_20: repository(owner: \"EFForg\", name: \"apkeep\") { ..._latest_release_tag }
_21: repository(owner: \"ximion\", name: \"appstream\") { ..._newest_tag }
_22: repository(owner: \"aptly-dev\", name: \"aptly\") { ..._latest_release_tag }
_23: repository(owner: \"argp-standalone\", name: \"argp-standalone\") { ..._newest_tag }
_24: repository(owner: \"aria2\", name: \"aria2\") { ..._newest_tag }
_25: repository(owner: \"opencollab\", name: \"arpack-ng\") { ..._newest_tag }
_26: repository(owner: \"asciidoc\", name: \"asciidoc-py3\") { ..._latest_release_tag }
_27: repository(owner: \"asciinema\", name: \"asciinema\") { ..._latest_release_tag }
_28: repository(owner: \"bergercookie\", name: \"asm-lsp\") { ..._newest_tag }
_29: repository(owner: \"assimp\", name: \"assimp\") { ..._latest_release_tag }
_30: repository(owner: \"OpenVisionE2\", name: \"astra-sm\") { ..._newest_tag }
_31: repository(owner: \"wez\", name: \"atomicparsley\") { ..._latest_release_tag }
_32: repository(owner: \"atomvm\", name: \"AtomVM\") { ..._latest_release_tag }
_33: repository(owner: \"ellie\", name: \"atuin\") { ..._latest_release_tag }
_34: repository(owner: \"wting\", name: \"autojump\") { ..._latest_release_tag }
_35: repository(owner: \"Ro5bert\", name: \"avra\") { ..._newest_tag }
_36: repository(owner: \"slavaGanzin\", name: \"await\") { ..._latest_release_tag }
_37: repository(owner: \"axel-download-accelerator\", name: \"axel\") { ..._latest_release_tag }
_38: repository(owner: \"BLAKE3-team\", name: \"BLAKE3\") { ..._latest_release_tag }
_39: repository(owner: \"fumiama\", name: \"base16384\") { ..._latest_release_tag }
_40: repository(owner: \"scop\", name: \"bash-completion\") { ..._latest_release_tag }
_41: repository(owner: \"sharkdp\", name: \"bat\") { ..._latest_release_tag }
_42: repository(owner: \"gavinhoward\", name: \"bc\") { ..._latest_release_tag }
_43: repository(owner: \"beanshell\", name: \"beanshell\") { ..._latest_release_tag }
_44: repository(owner: \"itchyny\", name: \"bed\") { ..._latest_release_tag }
_45: repository(owner: \"facebookincubator\", name: \"below\") { ..._newest_tag }
_46: repository(owner: \"rsharo\", name: \"bgrep\") { ..._latest_release_tag }
_47: repository(owner: \"louiz\", name: \"biboumi\") { ..._newest_tag }
_48: repository(owner: \"WebAssembly\", name: \"binaryen\") { ..._latest_release_tag }
_49: repository(owner: \"bitcoin\", name: \"bitcoin\") { ..._latest_release_tag }
_50: repository(owner: \"blade-lang\", name: \"blade\") { ..._latest_release_tag }
_51: repository(owner: \"jart\", name: \"blink\") { ..._latest_release_tag }
_52: repository(owner: \"blogc\", name: \"blogc\") { ..._latest_release_tag }
_53: repository(owner: \"tgraf\", name: \"bmon\") { ..._latest_release_tag }
_54: repository(owner: \"ekzhang\", name: \"bore\") { ..._latest_release_tag }
_55: repository(owner: \"borgbackup\", name: \"borg\") { ..._latest_release_tag }
_56: repository(owner: \"ascii-boxes\", name: \"boxes\") { ..._latest_release_tag }
_57: repository(owner: \"tmewett\", name: \"BrogueCE\") { ..._latest_release_tag }
_58: repository(owner: \"txthinking\", name: \"brook\") { ..._latest_release_tag }
_59: repository(owner: \"Canop\", name: \"broot\") { ..._latest_release_tag }
_60: repository(owner: \"google\", name: \"brotli\") { ..._latest_release_tag }
_61: repository(owner: \"bittorrent\", name: \"go-btfs\") { ..._latest_release_tag }
_62: repository(owner: \"bufbuild\", name: \"buf\") { ..._latest_release_tag }
_63: repository(owner: \"cabinpkg\", name: \"cabin\") { ..._latest_release_tag }
_64: repository(owner: \"caddyserver\", name: \"caddy\") { ..._latest_release_tag }
_65: repository(owner: \"lcn2\", name: \"calc\") { ..._latest_release_tag }
_66: repository(owner: \"c-ares\", name: \"c-ares\") { ..._latest_release_tag }
_67: repository(owner: \"karlstav\", name: \"cava\") { ..._latest_release_tag }
_68: repository(owner: \"kornelski\", name: \"cavif-rs\") { ..._latest_release_tag }
_69: repository(owner: \"ccache\", name: \"ccache\") { ..._latest_release_tag }
_70: repository(owner: \"0l1v3rr\", name: \"cli-file-manager\") { ..._latest_release_tag }
_71: repository(owner: \"CGAL\", name: \"cgal\") { ..._latest_release_tag }
_72: repository(owner: \"dloebl\", name: \"cgif\") { ..._latest_release_tag }
_73: repository(owner: \"hpjansson\", name: \"chafa\") { ..._latest_release_tag }
_74: repository(owner: \"libcheck\", name: \"check\") { ..._latest_release_tag }
_75: repository(owner: \"twpayne\", name: \"chezmoi\") { ..._latest_release_tag }
_76: repository(owner: \"theryangeary\", name: \"choose\") { ..._newest_tag }
_77: repository(owner: \"mitnk\", name: \"cicada\") { ..._latest_release_tag }
_78: repository(owner: \"CNugteren\", name: \"CLBlast\") { ..._latest_release_tag }
_79: repository(owner: \"leo-arch\", name: \"clifm\") { ..._latest_release_tag }
_80: repository(owner: \"Oblomov\", name: \"clinfo\") { ..._newest_tag }
_81: repository(owner: \"cloudflare\", name: \"cloudflared\") { ..._latest_release_tag }
_82: repository(owner: \"commonmark\", name: \"cmark\") { ..._latest_release_tag }
_83: repository(owner: \"abishekvashok\", name: \"cmatrix\") { ..._latest_release_tag }
_84: repository(owner: \"cmus\", name: \"cmus\") { ..._latest_release_tag }
_85: repository(owner: \"Arkq\", name: \"cmusfm\") { ..._newest_tag }
_86: repository(owner: \"exaexa\", name: \"codecrypt\") { ..._newest_tag }
_87: repository(owner: \"coin-or\", name: \"Cbc\") { ..._latest_release_tag }
_88: repository(owner: \"coin-or\", name: \"Clp\") { ..._latest_release_tag }
_89: repository(owner: \"cointop-sh\", name: \"cointop\") { ..._latest_release_tag }
_90: repository(owner: \"composer\", name: \"composer\") { ..._newest_tag }
_91: repository(owner: \"ros\", name: \"console_bridge\") { ..._latest_release_tag }
_92: repository(owner: \"cooklang\", name: \"cookcli\") { ..._latest_release_tag }
_93: repository(owner: \"DrakeW\", name: \"corgi\") { ..._latest_release_tag }
_94: repository(owner: \"cowsay-org\", name: \"cowsay\") { ..._latest_release_tag }
_95: repository(owner: \"danmar\", name: \"cppcheck\") { ..._latest_release_tag }
_96: repository(owner: \"Dr-Noob\", name: \"cpufetch\") { ..._latest_release_tag }
_97: repository(owner: \"opsengine\", name: \"cpulimit\") { ..._newest_tag }
_98: repository(owner: \"crawl\", name: \"crawl\") { ..._latest_release_tag }
_99: repository(owner: \"schollz\", name: \"croc\") { ..._latest_release_tag }
ratelimit: rateLimit { cost limit remaining used resetAt }
} |
a77d809
to
eacdeed
Compare
eacdeed
to
3be74ae
Compare
This PR aims to make the auto-update workflow rate-limit aware in order to avoid future shadowbans of @termuxbot2 for just doing its job.
In this current state it only implements basic debugging functionality by printing out the
.resources.core
table of the rate limit query API response which contains thelimit
(int),used
(int),remaining
(int) andreset
(unix_epoch) keys.For local testing I have generated a fine grained access token for GitHub, which I am sourcing from
gh.env
inside the root directory of the git repo.(This token only has read permissions to the rate-limit API endpoint, I may need to replace it for further testing)
The token is obviously passed in as part of the environment in the CI, so this file exists just for local testing.
Speaking of which, my local testing setup consists of running:
./scripts/run-docker.sh ./scripts/bin/update-packages '@all'
Which simulates a full auto update run.
(It also eats through the rate limit quite fast, which is why I'm opening this PR now, since I need to wait an hour to do more testing)