sep 11 for scrapy#629

sep/sep-011.rst

@@ -0,0 +1,36 @@
+=======  ================================
+SEP      11
+Title    Process models for Scrapy
+Created  2009-11-16
+Status   Partially implemented - see #168
+=======  ================================
+
+==================================
+SEP-011: Process models for Scrapy
+==================================
+
+There is an interest of supporting different process models for Scrapy, mainly
+to help prevent memory leaks which affect running all spiders in the same
+process.
+
+By running each spider on a separate process (or pool of processes) we'll be
+able to "recycle" process when they exceed a maximum amount of memory.
+
+Supported process models
+========================
+
+The user could choose between different process models:
+
+1. in process (only method supported so far)
+2. pooled processes (a predefined pool of N processes, which could run more than one spider each)
+3. separate processes (one process per spider)
+
+Using different processes would increase reliability at the cost of performance.
+
+Another ideas to consider
+=========================
+
+- configuring pipeline process models - so that we can have a process exclusive
+  for running pipelines
+- support writing spidersr in different languages when we don't use an in
+  process model