Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Worker hosts are unreachable #7

Open
sap9433 opened this issue May 11, 2018 · 0 comments
Open

Worker hosts are unreachable #7

sap9433 opened this issue May 11, 2018 · 0 comments

Comments

@sap9433
Copy link

sap9433 commented May 11, 2018

I started scrapyd , mongo and rabbitmq in worker threads still it shows worker status: unreachable , version: unknown

When I cal ping workers from Link generator curl working file

This is how ./scrapyproject/scrapy_packages/settings.py looks

`SCHEDULER = ".rabbitmq.scheduler.Scheduler"
SCHEDULER_PERSIST = True
RABBITMQ_HOST = '127.0.0.1'
RABBITMQ_PORT = 5672
RABBITMQ_USERNAME = 'guest'
RABBITMQ_PASSWORD = 'guest'

MONGODB_PUBLIC_ADDRESS = '127.0.0.1:27017' # This will be shown on the web intee
rface, but won't be used for connecting to DB
MONGODB_URI = '127.0.0.1:27017' # Actual uri to connect to DB
MONGODB_USER = ''
MONGODB_PASSWORD = ''
MONGODB_SHARDED = False
MONGODB_BUFFER_DATA = 100

LINK_GENERATOR = 'http://127.0.0.1:6800' # Set your link generator worker addree
ss here
SCRAPERS = ['172.17.0.4:3800', '172.17.0.2:6800'] # Set your scraper worker addd
resses here

LINUX_USER_CREATION_ENABLED = False # Set this to True if you want a linux userr
account created during registration`

And This is what I get when I curl them from link generator -

`root@96f3b9b83573:/usr/local/Distributed-Multi-User-Scrapy-System-with-a-Web-UI#
curl -I 172.17.0.4:3800
HTTP/1.1 200 OK
Date: Fri, 11 May 2018 04:50:31 GMT
Content-Length: 699
Content-Type: text/html
Server: TwistedWeb/18.4.0

root@96f3b9b83573:/usr/local/Distributed-Multi-User-Scrapy-System-with-a-Web-UI#
curl -I 172.17.0.2:6800
HTTP/1.1 200 OK
Date: Fri, 11 May 2018 04:50:50 GMT
Content-Length: 699
Content-Type: text/html
Server: TwistedWeb/18.4.0`

So workers are up .. but somehow link generator cant connect to them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant