New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[REFACTOR] Instagram 크롤링 멀티 스레드로 병렬 처리 #237

Open

moonyoungCHAE opened this issue Sep 16, 2020 · 0 comments

Labels

🛠 Refactoring

Collaborator

moonyoungCHAE commented Sep 16, 2020

리팩토링 대상 :

429 too many req 문제에서 proxy를 queue로 처리하면 429 문제가 해결되지 않을까 싶어서 시도해봤으면 좋겠음
proxy를 random으로 돌리다 보니 같은 proxy로 여러 번 req를 보낼 수도 있을 것 같음
queue로 돌리면 이런 경우를 방지할 수 있음
둔덩 말로 429가 나왔다가 안나왔다가 했다고 함

예상 공수 기간 :

특이 사항(optional) :

moonyoungCHAE added the 🛠 Refactoring label

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment