| SEP | 16 |
| Title | Leg Spider |
| Author | Insophia Team |
| Created | 2010-06-03 |
| Status | Superseded by :doc:`sep-018` |
This SEP introduces a new kind of Spider called LegSpider which provides
modular functionality which can be plugged to different spiders.
The purpose of Leg Spiders is to define an architecture for building spiders based on smaller well-tested components (aka. Legs) that can be combined to achieve the desired functionality. These reusable components will benefit all Scrapy users by building a repository of well-tested components (legs) that can be shared among different spiders and projects. Some of them will come bundled with Scrapy.
The Legs themselves can be also combined with sub-legs, in a hierarchical fashion. Legs are also spiders themselves, hence the name "Leg Spider".
A LegSpider is a BaseSpider subclass that adds the following attributes and methods:
legs- legs composing this spider
process_response(response)- Process a (downloaded) response and return a list of requests and items
process_request(request)- Process a request after it has been extracted and before returning it from the spider
process_item(item)- Process an item after it has been extracted and before returning it from the spider
set_spider()- Defines the main spider associated with this Leg Spider, which is often used to configure the Leg Spider behavior.
- Each Leg Spider has zero or many Leg Spiders associated with it. When a
response arrives, the Leg Spider process it with its
process_responsemethod and also theprocess_responsemethod of all its "sub leg spiders". Finally, the output of all of them is combined to produce the final aggregated output. - Each element of the aggregated output of
process_responseis processed with eitherprocess_itemorprocess_requestbefore being returned from the spider. Similar toprocess_response, each item/request is processed with allprocess_{request,item} of the leg spiders composing the spider, and also with those of the spider itself.
A typical application of LegSpider's is to build Link Extractors. For example:
#!python
class RegexHtmlLinkExtractor(LegSpider):
def process_response(self, response):
if isinstance(response, HtmlResponse):
allowed_regexes = self.spider.url_regexes_to_follow
# extract urls to follow using allowed_regexes
return [Request(x) for x in urls_to_follow]
class MySpider(LegSpider):
legs = [RegexHtmlLinkExtractor()]
url_regexes_to_follow = ['/product.php?.*']
def parse_response(self, response):
# parse response and extract items
return items
This is a Leg Spider that can be used for following links from RSS2 feeds.
#!python
class Rss2LinkExtractor(LegSpider):
def process_response(self, response):
if response.headers.get('Content-type') 'application/rss+xml':
xs = XmlXPathSelector(response)
urls = xs.select("//item/link/text()").extract()
return [Request(x) for x in urls]
Another example could be to build a callback dispatcher based on rules:
#!python
class CallbackRules(LegSpider):
def __init__(self, *a, **kw):
super(CallbackRules, self).__init__(*a, **kw)
for regex, method_name in self.spider.callback_rules.items():
r = re.compile(regex)
m = getattr(self.spider, method_name, None)
if m:
self._rules[r] = m
def process_response(self, response):
for regex, method in self._rules.items():
m = regex.search(response.url)
if m:
return method(response)
return []
class MySpider(LegSpider):
legs = [CallbackRules()]
callback_rules = {
'/product.php.*': 'parse_product',
'/category.php.*': 'parse_category',
}
def parse_product(self, response):
# parse response and populate item
return item
Another example could be for building URL canonicalizers:
#!python
class CanonializeUrl(LegSpider):
def process_request(self, request):
curl = canonicalize_url(request.url, rules=self.spider.canonicalization_rules)
return request.replace(url=curl)
class MySpider(LegSpider):
legs = [CanonicalizeUrl()]
canonicalization_rules = ['sort-query-args', 'normalize-percent-encoding', ...]
# ...
Another example could be for setting a unique identifier to items, based on certain fields:
#!python
class ItemIdSetter(LegSpider):
def process_item(self, item):
id_field = self.spider.id_field
id_fields_to_hash = self.spider.id_fields_to_hash
item[id_field] = make_hash_based_on_fields(item, id_fields_to_hash)
return item
class MySpider(LegSpider):
legs = [ItemIdSetter()]
id_field = 'guid'
id_fields_to_hash = ['supplier_name', 'supplier_id']
def process_response(self, item):
# extract item from response
return item
Here's an example that combines functionality from multiple leg spiders:
#!python
class MySpider(LegSpider):
legs = [RegexLinkExtractor(), ParseRules(), CanonicalizeUrl(), ItemIdSetter()]
url_regexes_to_follow = ['/product.php?.*']
parse_rules = {
'/product.php.*': 'parse_product',
'/category.php.*': 'parse_category',
}
canonicalization_rules = ['sort-query-args', 'normalize-percent-encoding', ...]
id_field = 'guid'
id_fields_to_hash = ['supplier_name', 'supplier_id']
def process_product(self, item):
# extract item from response
return item
def process_category(self, item):
# extract item from response
return item
A common question that would arise is when one should use Leg Spiders and when to use Spider middlewares. Leg Spiders functionality is meant to implement spider-specific functionality, like link extraction which has custom rules per spider. Spider middlewares, on the other hand, are meant to implement global functionality.
Leg Spiders are not a silver bullet to implement all kinds of spiders, so it's important to keep in mind their scope and limitations, such as:
- Leg Spiders can't filter duplicate requests, since they don't have access to all requests at the same time. This functionality should be done in a spider or scheduler middleware.
- Leg Spiders are meant to be used for spiders whose behavior (requests & items to extract) depends only on the current page and not previously crawled pages (aka. "context-free spiders"). If your spider has some custom logic with chained downloads (for example, multi-page items) then Leg Spiders may not be a good fit.
Here's a proof-of-concept implementation of LegSpider:
#!python
from scrapy.http import Request
from scrapy.item import BaseItem
from scrapy.spider import BaseSpider
from scrapy.utils.spider import iterate_spider_output
class LegSpider(BaseSpider):
"""A spider made of legs"""
legs = []
def __init__(self, *args, **kwargs):
super(LegSpider, self).__init__(*args, **kwargs)
self._legs = [self] + self.legs[:]
for l in self._legs:
l.set_spider(self)
def parse(self, response):
res = self._process_response(response)
for r in res:
if isinstance(r, BaseItem):
yield self._process_item(r)
else:
yield self._process_request(r)
def process_response(self, response):
return []
def process_request(self, request):
return request
def process_item(self, item):
return item
def set_spider(self, spider):
self.spider = spider
def _process_response(self, response):
res = []
for l in self._legs:
res.extend(iterate_spider_output(l.process_response(response)))
return res
def _process_request(self, request):
for l in self._legs:
request = l.process_request(request)
return request
def _process_item(self, item):
for l in self._legs:
item = l.process_item(item)
return item