converted sep 16

sep/sep-016.rst

@@ -0,0 +1,306 @@
+=======  =============================
+SEP      16
+Title    Leg Spider
+Author   Insophia Team
+Created  2010-06-03
+Status   Superseded  by :doc:`sep-018`
+=======  =============================
+
+===================
+SEP-016: Leg Spider
+===================
+
+This SEP introduces a new kind of Spider called ``LegSpider`` which provides
+modular functionality which can be plugged to different spiders.
+
+Rationale
+=========
+
+The purpose of Leg Spiders is to define an architecture for building spiders
+based on smaller well-tested components (aka. Legs) that can be combined to
+achieve the desired functionality. These reusable components will benefit all
+Scrapy users by building a repository of well-tested components (legs) that can
+be shared among different spiders and projects. Some of them will come bundled
+with Scrapy.
+
+The Legs themselves can be also combined with sub-legs, in a hierarchical
+fashion. Legs are also spiders themselves, hence the name "Leg Spider".
+
+``LegSpider`` API
+=================
+
+A ``LegSpider`` is a ``BaseSpider`` subclass that adds the following attributes and methods:
+
+- ``legs``
+   - legs composing this spider
+- ``process_response(response)``
+   - Process a (downloaded) response and return a list of requests and items
+- ``process_request(request)``
+   - Process a request after it has been extracted and before returning it from
+     the spider
+- ``process_item(item)``
+   - Process an item after it has been extracted and before returning it from
+     the spider
+- ``set_spider()``
+   - Defines the main spider associated with this Leg Spider, which is often
+     used to configure the Leg Spider behavior.
+
+How Leg Spiders work
+====================
+
+1. Each Leg Spider has zero or many Leg Spiders associated with it. When a
+   response arrives, the Leg Spider process it with its ``process_response``
+   method and also the ``process_response`` method of all its "sub leg
+   spiders". Finally, the output of all of them is combined to produce the
+   final aggregated output.
+2. Each element of the aggregated output of ``process_response`` is processed
+   with either ``process_item`` or ``process_request`` before being returned
+   from the spider. Similar to ``process_response``, each item/request is
+   processed with all ``process_{request,item``} of the leg spiders composing
+   the spider, and also with those of the spider itself.
+
+Leg Spider examples
+===================
+
+Regex (HTML) Link Extractor
+---------------------------
+
+A typical application of LegSpider's is to build Link Extractors. For example:
+
+::
+
+   #!python
+   class RegexHtmlLinkExtractor(LegSpider):
+
+       def process_response(self, response):
+           if isinstance(response, HtmlResponse):
+               allowed_regexes = self.spider.url_regexes_to_follow
+               # extract urls to follow using allowed_regexes
+               return [Request(x) for x in urls_to_follow]
+
+   class MySpider(LegSpider):
+
+       legs = [RegexHtmlLinkExtractor()]
+       url_regexes_to_follow = ['/product.php?.*']
+
+       def parse_response(self, response):
+           # parse response and extract items
+           return items
+
+RSS2 link extractor
+-------------------
+
+This is a Leg Spider that can be used for following links from RSS2 feeds.
+
+::
+
+   #!python
+   class Rss2LinkExtractor(LegSpider):
+
+       def process_response(self, response):
+           if response.headers.get('Content-type') 'application/rss+xml':
+               xs = XmlXPathSelector(response)
+               urls = xs.select("//item/link/text()").extract()
+               return [Request(x) for x in urls]
+
+Callback dispatcher based on rules
+----------------------------------
+
+Another example could be to build a callback dispatcher based on rules:
+
+::
+
+   #!python
+   class CallbackRules(LegSpider):
+
+       def __init__(self, *a, **kw):
+           super(CallbackRules, self).__init__(*a, **kw)
+           for regex, method_name in self.spider.callback_rules.items():
+               r = re.compile(regex)
+               m = getattr(self.spider, method_name, None)
+               if m:
+                   self._rules[r] = m
+
+       def process_response(self, response):
+           for regex, method in self._rules.items():
+               m = regex.search(response.url)
+               if m:
+                   return method(response)
+           return []
+
+   class MySpider(LegSpider):
+
+       legs = [CallbackRules()]
+       callback_rules = {
+           '/product.php.*': 'parse_product',
+           '/category.php.*': 'parse_category',
+       }
+
+       def parse_product(self, response):
+           # parse response and populate item
+           return item
+
+URL Canonicalizers
+------------------
+
+Another example could be for building URL canonicalizers:
+
+::
+
+   #!python
+   class CanonializeUrl(LegSpider):
+
+       def process_request(self, request):
+           curl = canonicalize_url(request.url, rules=self.spider.canonicalization_rules)
+           return request.replace(url=curl)
+
+   class MySpider(LegSpider):
+
+       legs = [CanonicalizeUrl()]
+       canonicalization_rules = ['sort-query-args', 'normalize-percent-encoding', ...]
+
+       # ...
+
+Setting item identifier
+-----------------------
+
+Another example could be for setting a unique identifier to items, based on
+certain fields:
+
+::
+
+   #!python
+   class ItemIdSetter(LegSpider):
+
+       def process_item(self, item):
+           id_field = self.spider.id_field
+           id_fields_to_hash = self.spider.id_fields_to_hash
+           item[id_field] = make_hash_based_on_fields(item, id_fields_to_hash)
+           return item
+
+   class MySpider(LegSpider):
+
+       legs = [ItemIdSetter()]
+       id_field = 'guid'
+       id_fields_to_hash = ['supplier_name', 'supplier_id']
+
+       def process_response(self, item):
+           # extract item from response
+           return item
+
+Combining multiple leg spiders
+------------------------------
+
+Here's an example that combines functionality from multiple leg spiders:
+
+::
+
+   #!python
+   class MySpider(LegSpider):
+
+       legs = [RegexLinkExtractor(), ParseRules(), CanonicalizeUrl(), ItemIdSetter()]
+
+       url_regexes_to_follow = ['/product.php?.*']
+
+       parse_rules = {
+           '/product.php.*': 'parse_product',
+           '/category.php.*': 'parse_category',
+       }
+
+       canonicalization_rules = ['sort-query-args', 'normalize-percent-encoding', ...]
+
+       id_field = 'guid'
+       id_fields_to_hash = ['supplier_name', 'supplier_id']
+
+       def process_product(self, item):
+           # extract item from response
+           return item
+
+       def process_category(self, item):
+           # extract item from response
+           return item
+
+Leg Spiders vs Spider middlewares
+=================================
+
+A common question that would arise is when one should use Leg Spiders and when
+to use Spider middlewares. Leg Spiders functionality is meant to implement
+spider-specific functionality, like link extraction which has custom rules per
+spider. Spider middlewares, on the other hand, are meant to implement global
+functionality.
+
+When not to use Leg Spiders
+===========================
+
+Leg Spiders are not a silver bullet to implement all kinds of spiders, so it's
+important to keep in mind their scope and limitations, such as:
+
+- Leg Spiders can't filter duplicate requests, since they don't have access to
+  all requests at the same time. This functionality should be done in a spider
+  or scheduler middleware.
+- Leg Spiders are meant to be used for spiders whose behavior (requests & items
+  to extract) depends only on the current page and not previously crawled pages
+  (aka. "context-free spiders"). If your spider has some custom logic with
+  chained downloads (for example, multi-page items) then Leg Spiders may not be
+  a good fit.
+
+``LegSpider`` proof-of-concept implementation
+=============================================
+
+Here's a proof-of-concept implementation of ``LegSpider``:
+
+::
+
+   #!python
+   from scrapy.http import Request
+   from scrapy.item import BaseItem
+   from scrapy.spider import BaseSpider
+   from scrapy.utils.spider import iterate_spider_output
+
+
+   class LegSpider(BaseSpider):
+       """A spider made of legs"""
+
+       legs = []
+
+       def __init__(self, *args, **kwargs):
+           super(LegSpider, self).__init__(*args, **kwargs)
+           self._legs = [self] + self.legs[:]
+           for l in self._legs:
+               l.set_spider(self)
+
+       def parse(self, response):
+           res = self._process_response(response)
+           for r in res:
+               if isinstance(r, BaseItem):
+                   yield self._process_item(r)
+               else:
+                   yield self._process_request(r)
+
+       def process_response(self, response):
+           return []
+
+       def process_request(self, request):
+           return request
+
+       def process_item(self, item):
+           return item
+
+       def set_spider(self, spider):
+           self.spider = spider
+
+       def _process_response(self, response):
+           res = []
+           for l in self._legs:
+               res.extend(iterate_spider_output(l.process_response(response)))
+           return res
+
+       def _process_request(self, request):
+           for l in self._legs:
+               request = l.process_request(request)
+           return request
+
+       def _process_item(self, item):
+           for l in self._legs:
+               item = l.process_item(item)
+           return item