Writing Spiders

For now, we recommend looking at the following spider definitions to get a feel for writing them:

• spiders/thepioneerwoman_spider.py
• spiders/thepioneerwoman_feedspider.py

Both files are extensively documented, and should give you an idea of what's involved. If you have questions, check the Feedback section and hit us up.

To generate your own spider, use the included generate.py program. From the scrapy_proj directory, run the following (make sure you are in the correct virtualenv:

python generate.py SPIDER_NAME START_URL

This will generate a basic spider for you named SPIDER_NAME that starts crawling at START_URL. All that remains for you to do is to fill in the correct info for scraping the name, image, etc. See `python generate.py --help' for other command line options.

Please follow "PEP 8 - Style Guide for Python Code" for code you write.