New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Explore full scrape optimizations #37

Open

antidipyramid opened this issue Jan 21, 2025 · 0 comments

Assignees

Collaborator

antidipyramid commented Jan 21, 2025

Full scrapes are verging on 16 hours-- we might be able to split up scrape jobs (e.g. by year).

(Might need to up the # of concurrent processes allowed in the Dashboard)

antidipyramid self-assigned this

github-project-automation bot added this to boardagendas.metro.net - Monthly priorities

github-project-automation bot moved this to 🆕 New issues in boardagendas.metro.net - Monthly priorities

antidipyramid moved this from 🆕 New issues to 📥 Backlog in boardagendas.metro.net - Monthly priorities

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment