Skip to content

Commit

Permalink
Switch from to ` where inline code formatting is desired
Browse files Browse the repository at this point in the history
  • Loading branch information
Gallaecio committed Mar 1, 2019
1 parent c72ab1d commit 75d6f56
Show file tree
Hide file tree
Showing 27 changed files with 148 additions and 147 deletions.
2 changes: 1 addition & 1 deletion docs/contributing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ guidelines when you're going to report a new bug.

* search the `scrapy-users`_ list and `Scrapy subreddit`_ to see if it has
been discussed there, or if you're not sure if what you're seeing is a bug.
You can also ask in the `#scrapy` IRC channel.
You can also ask in the ``#scrapy`` IRC channel.

* write **complete, reproducible, specific bug reports**. The smaller the test
case, the better. Remember that other developers won't have your project to
Expand Down
130 changes: 65 additions & 65 deletions docs/news.rst

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions docs/topics/api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ how you :ref:`configure the downloader middlewares
.. method:: crawl(\*args, \**kwargs)

Starts the crawler by instantiating its spider class with the given
`args` and `kwargs` arguments, while setting the execution engine in
``args`` and ``kwargs`` arguments, while setting the execution engine in
motion.

Returns a deferred that is fired when the crawl is finished.
Expand Down Expand Up @@ -180,7 +180,7 @@ SpiderLoader API
.. method:: load(spider_name)

Get the Spider class with the given name. It'll look into the previously
loaded spiders for a spider class with name `spider_name` and will raise
loaded spiders for a spider class with name ``spider_name`` and will raise
a KeyError if not found.

:param spider_name: spider class name
Expand Down
40 changes: 20 additions & 20 deletions docs/topics/downloader-middleware.rst
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ previous (or subsequent) middleware being applied.

If you want to disable a built-in middleware (the ones defined in
:setting:`DOWNLOADER_MIDDLEWARES_BASE` and enabled by default) you must define it
in your project's :setting:`DOWNLOADER_MIDDLEWARES` setting and assign `None`
in your project's :setting:`DOWNLOADER_MIDDLEWARES` setting and assign ``None``
as its value. For example, if you want to disable the user-agent middleware::

DOWNLOADER_MIDDLEWARES = {
Expand Down Expand Up @@ -357,7 +357,7 @@ HttpCacheMiddleware

.. reqmeta:: dont_cache

You can also avoid caching a response on every policy using :reqmeta:`dont_cache` meta key equals `True`.
You can also avoid caching a response on every policy using :reqmeta:`dont_cache` meta key equals ``True``.

.. _httpcache-policy-dummy:

Expand Down Expand Up @@ -390,33 +390,33 @@ runs to avoid downloading unmodified data (to save bandwidth and speed up crawls

what is implemented:

* Do not attempt to store responses/requests with `no-store` cache-control directive set
* Do not serve responses from cache if `no-cache` cache-control directive is set even for fresh responses
* Compute freshness lifetime from `max-age` cache-control directive
* Compute freshness lifetime from `Expires` response header
* Compute freshness lifetime from `Last-Modified` response header (heuristic used by Firefox)
* Compute current age from `Age` response header
* Compute current age from `Date` header
* Revalidate stale responses based on `Last-Modified` response header
* Revalidate stale responses based on `ETag` response header
* Set `Date` header for any received response missing it
* Support `max-stale` cache-control directive in requests
* Do not attempt to store responses/requests with ``no-store`` cache-control directive set
* Do not serve responses from cache if ``no-cache`` cache-control directive is set even for fresh responses
* Compute freshness lifetime from ``max-age`` cache-control directive
* Compute freshness lifetime from ``Expires`` response header
* Compute freshness lifetime from ``Last-Modified`` response header (heuristic used by Firefox)
* Compute current age from ``Age`` response header
* Compute current age from ``Date`` header
* Revalidate stale responses based on ``Last-Modified`` response header
* Revalidate stale responses based on ``ETag`` response header
* Set ``Date`` header for any received response missing it
* Support ``max-stale`` cache-control directive in requests

This allows spiders to be configured with the full RFC2616 cache policy,
but avoid revalidation on a request-by-request basis, while remaining
conformant with the HTTP spec.

Example:

Add `Cache-Control: max-stale=600` to Request headers to accept responses that
Add ``Cache-Control: max-stale=600`` to Request headers to accept responses that
have exceeded their expiration time by no more than 600 seconds.

See also: RFC2616, 14.9.3

what is missing:

* `Pragma: no-cache` support https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.1
* `Vary` header support https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.6
* ``Pragma: no-cache`` support https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.1
* ``Vary`` header support https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.6
* Invalidation after updates or deletes https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.10
* ... probably others ..

Expand Down Expand Up @@ -626,12 +626,12 @@ Default: ``False``
If enabled, will cache pages unconditionally.

A spider may wish to have all responses available in the cache, for
future use with `Cache-Control: max-stale`, for instance. The
future use with ``Cache-Control: max-stale``, for instance. The
DummyPolicy caches all responses but never revalidates them, and
sometimes a more nuanced policy is desirable.

This setting still respects `Cache-Control: no-store` directives in responses.
If you don't want that, filter `no-store` out of the Cache-Control headers in
This setting still respects ``Cache-Control: no-store`` directives in responses.
If you don't want that, filter ``no-store`` out of the Cache-Control headers in
responses you feedto the cache middleware.

.. setting:: HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLS
Expand Down Expand Up @@ -940,7 +940,7 @@ UserAgentMiddleware

Middleware that allows spiders to override the default user agent.

In order for a spider to override the default user agent, its `user_agent`
In order for a spider to override the default user agent, its ``user_agent``
attribute must be set.

.. _ajaxcrawl-middleware:
Expand Down
2 changes: 1 addition & 1 deletion docs/topics/exporters.rst
Original file line number Diff line number Diff line change
Expand Up @@ -303,7 +303,7 @@ CsvItemExporter

The additional keyword arguments of this constructor are passed to the
:class:`BaseItemExporter` constructor, and the leftover arguments to the
`csv.writer`_ constructor, so you can use any `csv.writer` constructor
`csv.writer`_ constructor, so you can use any ``csv.writer`` constructor
argument to customize this exporter.

A typical output of this exporter would be::
Expand Down
4 changes: 2 additions & 2 deletions docs/topics/extensions.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ settings, just like any other Scrapy code.
It is customary for extensions to prefix their settings with their own name, to
avoid collision with existing (and future) extensions. For example, a
hypothetic extension to handle `Google Sitemaps`_ would use settings like
`GOOGLESITEMAP_ENABLED`, `GOOGLESITEMAP_DEPTH`, and so on.
``GOOGLESITEMAP_ENABLED``, ``GOOGLESITEMAP_DEPTH``, and so on.

.. _Google Sitemaps: https://en.wikipedia.org/wiki/Sitemaps

Expand Down Expand Up @@ -368,7 +368,7 @@ Invokes a `Python debugger`_ inside a running Scrapy process when a `SIGUSR2`_
signal is received. After the debugger is exited, the Scrapy process continues
running normally.

For more info see `Debugging in Python`.
For more info see `Debugging in Python`_.

This extension only works on POSIX-compliant platforms (ie. not Windows).

Expand Down
2 changes: 1 addition & 1 deletion docs/topics/jobs.rst
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ on cookies.
Request serialization
---------------------

Requests must be serializable by the `pickle` module, in order for persistence
Requests must be serializable by the ``pickle`` module, in order for persistence
to work, so you should make sure that your requests are serializable.

The most common issue here is to use ``lambda`` functions on request callbacks that
Expand Down
2 changes: 1 addition & 1 deletion docs/topics/loaders.rst
Original file line number Diff line number Diff line change
Expand Up @@ -286,7 +286,7 @@ ItemLoader objects
given, one is instantiated automatically using the class in
:attr:`default_item_class`.

When instantiated with a `selector` or a `response` parameters
When instantiated with a ``selector`` or a ``response`` parameters
the :class:`ItemLoader` class provides convenient mechanisms for extracting
data from web pages using :ref:`selectors <topics-selectors>`.

Expand Down
2 changes: 1 addition & 1 deletion docs/topics/logging.rst
Original file line number Diff line number Diff line change
Expand Up @@ -243,7 +243,7 @@ scrapy.utils.log module
case, its usage is not required but it's recommended.

If you plan on configuring the handlers yourself is still recommended you
call this function, passing `install_root_handler=False`. Bear in mind
call this function, passing ``install_root_handler=False``. Bear in mind
there won't be any log output set by default in that case.

To get you started on manually configuring logging's output, you can use
Expand Down
2 changes: 1 addition & 1 deletion docs/topics/media-pipeline.rst
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ For example, the following image URL::

http://www.example.com/image.jpg

Whose `SHA1 hash` is::
Whose ``SHA1 hash`` is::

3afec3b4765f8f0a07b78f98c07b83f013567a0a

Expand Down
2 changes: 1 addition & 1 deletion docs/topics/practices.rst
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ returned by the :meth:`CrawlerRunner.crawl
<scrapy.crawler.CrawlerRunner.crawl>` method.

Here's an example of its usage, along with a callback to manually stop the
reactor after `MySpider` has finished running.
reactor after ``MySpider`` has finished running.

::

Expand Down
8 changes: 4 additions & 4 deletions docs/topics/request-response.rst
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ Request objects
:type meta: dict

:param body: the request body. If a ``unicode`` is passed, then it's encoded to
``str`` using the `encoding` passed (which defaults to ``utf-8``). If
``str`` using the ``encoding`` passed (which defaults to ``utf-8``). If
``body`` is not given, an empty string is stored. Regardless of the
type of this argument, the final value stored will be a ``str`` (never
``unicode`` or ``None``).
Expand Down Expand Up @@ -610,7 +610,7 @@ Response objects
.. attribute:: Response.flags

A list that contains flags for this response. Flags are labels used for
tagging Responses. For example: `'cached'`, `'redirected`', etc. And
tagging Responses. For example: ``'cached'``, ``'redirected``', etc. And
they're shown on the string representation of the Response (`__str__`
method) which is used by the engine for logging.

Expand Down Expand Up @@ -682,15 +682,15 @@ TextResponse objects

``unicode(response.body)`` is not a correct way to convert response
body to unicode: you would be using the system default encoding
(typically `ascii`) instead of the response encoding.
(typically ``ascii``) instead of the response encoding.


.. attribute:: TextResponse.encoding

A string with the encoding of this response. The encoding is resolved by
trying the following mechanisms, in order:

1. the encoding passed in the constructor `encoding` argument
1. the encoding passed in the constructor ``encoding`` argument

2. the encoding declared in the Content-Type HTTP header. If this
encoding is not valid (ie. unknown), it is ignored and the next
Expand Down
2 changes: 1 addition & 1 deletion docs/topics/selectors.rst
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ Constructing from response - :class:`~scrapy.http.HtmlResponse` is one of
Using selectors
---------------

To explain how to use the selectors we'll use the `Scrapy shell` (which
To explain how to use the selectors we'll use the ``Scrapy shell`` (which
provides interactive testing) and an example page located in the Scrapy
documentation server:

Expand Down
4 changes: 2 additions & 2 deletions docs/topics/settings.rst
Original file line number Diff line number Diff line change
Expand Up @@ -599,7 +599,7 @@ The amount of time (in secs) that the downloader will wait before timing out.
DOWNLOAD_MAXSIZE
----------------

Default: `1073741824` (1024MB)
Default: ``1073741824`` (1024MB)

The maximum response size (in bytes) that downloader will download.

Expand All @@ -620,7 +620,7 @@ If you want to disable it set to 0.
DOWNLOAD_WARNSIZE
-----------------

Default: `33554432` (32MB)
Default: ``33554432`` (32MB)

The response size (in bytes) that downloader will start to warn.

Expand Down
4 changes: 2 additions & 2 deletions docs/topics/spider-middleware.rst
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ previous (or subsequent) middleware being applied.

If you want to disable a builtin middleware (the ones defined in
:setting:`SPIDER_MIDDLEWARES_BASE`, and enabled by default) you must define it
in your project :setting:`SPIDER_MIDDLEWARES` setting and assign `None` as its
in your project :setting:`SPIDER_MIDDLEWARES` setting and assign ``None`` as its
value. For example, if you want to disable the off-site middleware::

SPIDER_MIDDLEWARES = {
Expand Down Expand Up @@ -200,7 +200,7 @@ DepthMiddleware
.. class:: DepthMiddleware

DepthMiddleware is used for tracking the depth of each Request inside the
site being scraped. It works by setting `request.meta['depth'] = 0` whenever
site being scraped. It works by setting ``request.meta['depth'] = 0`` whenever
there is no value previously set (usually just the first Request) and
incrementing it by 1 otherwise.

Expand Down
6 changes: 3 additions & 3 deletions docs/topics/spiders.rst
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ scrapy.Spider

You probably won't need to override this directly because the default
implementation acts as a proxy to the :meth:`__init__` method, calling
it with the given arguments `args` and named arguments `kwargs`.
it with the given arguments ``args`` and named arguments ``kwargs``.

Nonetheless, this method sets the :attr:`crawler` and :attr:`settings`
attributes in the new instance so they can be accessed later inside the
Expand Down Expand Up @@ -298,13 +298,13 @@ The above example can also be written as follows::

Keep in mind that spider arguments are only strings.
The spider will not do any parsing on its own.
If you were to set the `start_urls` attribute from the command line,
If you were to set the ``start_urls`` attribute from the command line,
you would have to parse it on your own into a list
using something like
`ast.literal_eval <https://docs.python.org/library/ast.html#ast.literal_eval>`_
or `json.loads <https://docs.python.org/library/json.html#json.loads>`_
and then set it as an attribute.
Otherwise, you would cause iteration over a `start_urls` string
Otherwise, you would cause iteration over a ``start_urls`` string
(a very common python pitfall)
resulting in each character being seen as a separate url.

Expand Down
4 changes: 2 additions & 2 deletions docs/topics/ubuntu.rst
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ To use the packages:

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 627220E7

2. Create `/etc/apt/sources.list.d/scrapy.list` file using the following command::
2. Create ``/etc/apt/sources.list.d/scrapy.list`` file using the following command::

echo 'deb http://archive.scrapy.org/ubuntu scrapy main' | sudo tee /etc/apt/sources.list.d/scrapy.list

Expand All @@ -34,7 +34,7 @@ To use the packages:
.. note:: Repeat step 3 if you are trying to upgrade Scrapy.

.. warning:: `python-scrapy` is a different package provided by official debian
.. warning:: ``python-scrapy`` is a different package provided by official debian
repositories, it's very outdated and it isn't supported by Scrapy team.

.. _Scrapinghub: https://scrapinghub.com/
Expand Down
10 changes: 5 additions & 5 deletions scrapy/crawler.py
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ def crawl(self, crawler_or_spidercls, *args, **kwargs):
It will call the given Crawler's :meth:`~Crawler.crawl` method, while
keeping track of it so it can be stopped later.
If `crawler_or_spidercls` isn't a :class:`~scrapy.crawler.Crawler`
If ``crawler_or_spidercls`` isn't a :class:`~scrapy.crawler.Crawler`
instance, this method will try to create one using this parameter as
the spider class given to it.
Expand Down Expand Up @@ -188,10 +188,10 @@ def create_crawler(self, crawler_or_spidercls):
"""
Return a :class:`~scrapy.crawler.Crawler` object.
* If `crawler_or_spidercls` is a Crawler, it is returned as-is.
* If `crawler_or_spidercls` is a Spider subclass, a new Crawler
* If ``crawler_or_spidercls`` is a Crawler, it is returned as-is.
* If ``crawler_or_spidercls`` is a Spider subclass, a new Crawler
is constructed for it.
* If `crawler_or_spidercls` is a string, this function finds
* If ``crawler_or_spidercls`` is a string, this function finds
a spider with this name in a Scrapy project (using spider loader),
then creates a Crawler instance for it.
"""
Expand Down Expand Up @@ -273,7 +273,7 @@ def start(self, stop_after_crawl=True):
:setting:`REACTOR_THREADPOOL_MAXSIZE`, and installs a DNS cache based
on :setting:`DNSCACHE_ENABLED` and :setting:`DNSCACHE_SIZE`.
If `stop_after_crawl` is True, the reactor will be stopped after all
If ``stop_after_crawl`` is True, the reactor will be stopped after all
crawlers have finished, using :meth:`join`.
:param boolean stop_after_crawl: stop or not the reactor when all
Expand Down
18 changes: 9 additions & 9 deletions scrapy/logformatter.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,21 +13,21 @@
class LogFormatter(object):
"""Class for generating log messages for different actions.
All methods must return a dictionary listing the parameters `level`, `msg`
and `args` which are going to be used for constructing the log message when
calling logging.log.
All methods must return a dictionary listing the parameters ``level``,
``msg`` and ``args`` which are going to be used for constructing the log
message when calling logging.log.
Dictionary keys for the method outputs:
* `level` should be the log level for that action, you can use those
* ``level`` should be the log level for that action, you can use those
from the python logging library: logging.DEBUG, logging.INFO,
logging.WARNING, logging.ERROR and logging.CRITICAL.
* `msg` should be a string that can contain different formatting
placeholders. This string, formatted with the provided `args`, is going
to be the log message for that action.
* ``msg`` should be a string that can contain different formatting
placeholders. This string, formatted with the provided ``args``, is
going to be the log message for that action.
* `args` should be a tuple or dict with the formatting placeholders for
`msg`. The final log message is computed as output['msg'] %
* ``args`` should be a tuple or dict with the formatting placeholders
for ``msg``. The final log message is computed as output['msg'] %
output['args'].
"""

Expand Down
6 changes: 3 additions & 3 deletions scrapy/pipelines/files.py
Original file line number Diff line number Diff line change
Expand Up @@ -255,13 +255,13 @@ class FilesPipeline(MediaPipeline):
doing stat of the files and determining if file is new, uptodate or
expired.
`new` files are those that pipeline never processed and needs to be
``new`` files are those that pipeline never processed and needs to be
downloaded from supplier site the first time.
`uptodate` files are the ones that the pipeline processed and are still
``uptodate`` files are the ones that the pipeline processed and are still
valid files.
`expired` files are those that pipeline already processed but the last
``expired`` files are those that pipeline already processed but the last
modification was made long time ago, so a reprocessing is recommended to
refresh it in case of change.
Expand Down
Loading

0 comments on commit 75d6f56

Please sign in to comment.