Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge latest master #8

Draft
wants to merge 159 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
159 commits
Select commit Hold shift + click to select a range
10f5d96
Upgrade asis.
Oct 19, 2016
37d7e3f
Update url-py.
Oct 19, 2016
493eef4
Drop the use of requests.Session.
Oct 10, 2016
0405267
Add a directive class.
Oct 10, 2016
2a89ed5
Rebase me into failing directive test.
Oct 13, 2016
e398e13
Rebase me into passing directive test.
Oct 13, 2016
d55b97c
Failing test about util.pairs.
Oct 13, 2016
055ddea
Passing test about util.pairs.
Oct 13, 2016
f7cc895
Failing tests about an Agent class.
Oct 13, 2016
0344ee5
Passing tests about an Agent class.
Oct 13, 2016
dc224cf
Failing tests about util.parse_date.
Oct 13, 2016
6caea62
Passing tests about util.parse_date.
Oct 13, 2016
e46adf5
Failing test about ttl policies.
Oct 13, 2016
e1ffc52
Passing test about ttl policies.
Oct 13, 2016
6e3023e
Failing tests about robots.txt.
Oct 14, 2016
510ffd8
Passing tests about robots.txt.
Oct 14, 2016
d1ac48d
Slash and burn.
Oct 13, 2016
be2e1e3
Update bench.py.
Oct 19, 2016
2293511
Require 100% coverage.
Oct 19, 2016
56903f5
Drop pypi and python 3 in Travis, temporarily.
Oct 28, 2016
8c77dfc
Merge pull request #32 from seomoz/dan/modernize
Oct 28, 2016
9ffd69e
Add dependency on rep-cpp. Rebase me once this revision lands on master.
Nov 1, 2016
55709d0
Add vagrant provisioning.
Oct 31, 2016
12a655a
Add dev dependency on Cython.
Nov 1, 2016
dffff12
Add dependency on six.
Nov 3, 2016
82684d4
Use rep-cpp.
Nov 1, 2016
5b5d7f7
Update README.
Nov 3, 2016
752f350
Start testing against python 3 again.
Nov 3, 2016
e05e3a5
Version bump to 0.4.0.
Nov 1, 2016
d65bd0f
Merge pull request #33 from seomoz/dan/rep-cpp
Nov 4, 2016
6647ed0
Expose .expires on Robots objects.
Nov 5, 2016
7ddfac9
Add cachetools as a dependency.
Nov 5, 2016
3e0e86d
Remove dependency on asis, unify dev-requirements{,-py3}.txt
Nov 7, 2016
ab74001
AgentCache and RobotsCache.
Nov 7, 2016
3646452
Version bump to 0.4.1.
Nov 5, 2016
ba367e5
Merge pull request #34 from seomoz/dan/big-4194
Nov 7, 2016
2713b5c
MANIFEST.in includes extension source files.
Nov 8, 2016
4a50aa6
Merge pull request #35 from seomoz/dan/add-manifest
Nov 8, 2016
d0f8458
Failing test about query-only directives.
Nov 10, 2016
087ee05
Passing test about query-only directives.
Nov 10, 2016
d8bf6c5
Version bump to 0.4.2.
Nov 10, 2016
28fc42b
Merge pull request #36 from seomoz/dan/query-only-directive
Nov 11, 2016
abbb222
Failing test about robots.txt agent case insensitivity.
Nov 16, 2016
419135c
Passing test about robots.txt agent case insensitivity. Rebase me whe…
Nov 16, 2016
e9e2423
Version bump to 0.4.3.
Nov 16, 2016
a8b0106
Merge pull request #38 from seomoz/dan/case-insensitive-user
Nov 16, 2016
2d163b7
Add missing .python-version.
b4hand Dec 7, 2016
fd1fc9c
Merge pull request #40 from seomoz/cbf/add-missing-python-version
b4hand Dec 7, 2016
6a5f92c
Fix docs to use right method name.
b4hand Dec 7, 2016
d0cdcc3
Merge pull request #41 from seomoz/cbf/update-docs
b4hand Dec 7, 2016
1aaf69e
Drop trailing whitespace.
b4hand Jan 18, 2017
b1dc7e6
Bump rep-cpp to latest version.
b4hand Jan 19, 2017
82f4beb
Merge pull request #49 from seomoz/cbf/bump-rep-cpp
b4hand Jan 19, 2017
b3cf040
Fix handling of Cache-Control header.
b4hand Feb 7, 2017
60b3367
Bump reppy version.
b4hand Feb 7, 2017
813baf5
Merge pull request #51 from seomoz/cbf/fix-cache-control
b4hand Feb 7, 2017
1ce8080
Update docs to include `--tags` option to push release tags.
b4hand Feb 7, 2017
c643869
Merge pull request #52 from seomoz/cbf/tags-docs
b4hand Feb 7, 2017
42345b6
Drop unused import.
b4hand Feb 8, 2017
11a36d0
Merge pull request #53 from seomoz/cbf/drop-unused-import
b4hand Feb 8, 2017
6d0379b
Bump rep-cpp version.
b4hand Feb 8, 2017
f407443
Bump reppy version.
b4hand Feb 8, 2017
aa1f7d2
Fix test from rep-cpp change.
b4hand Feb 8, 2017
36f37e8
Add additional assertions to test case.
b4hand Feb 8, 2017
71fbf18
Merge pull request #54 from seomoz/cbf/bump-rep-cpp-version
b4hand Feb 9, 2017
aa52ce9
Drop unused imports.
b4hand Feb 9, 2017
b1d073d
Merge pull request #55 from seomoz/cbf/drop-unused-imports
b4hand Feb 9, 2017
0b23925
Add failing test for invalid port.
b4hand Feb 13, 2017
a310c28
Handle invalid port numbers and other exceptions from robotsUrl.
b4hand Feb 13, 2017
524b197
Bump to version 0.4.6.
b4hand Feb 13, 2017
420b3bf
Merge pull request #58 from seomoz/cbf/handle-invalid-port
b4hand Feb 13, 2017
1e4c934
PEP-8 changes for setup.py.
b4hand Feb 13, 2017
2fef49c
Merge pull request #59 from seomoz/cbf/pep-8-setup-py
b4hand Feb 13, 2017
db9ef95
Automatically activate virtualenv on `vagrant ssh`.
b4hand Mar 3, 2017
e68ae72
Add pip install -r dev-requirements.txt
b4hand Mar 3, 2017
4b22908
Merge pull request #61 from seomoz/cbf/simplify-vagrant-env
b4hand Mar 3, 2017
2ac89f0
Drop url-py as dependency.
b4hand Mar 3, 2017
73e0142
Merge pull request #62 from seomoz/cbf/drop-url-py-dependency
b4hand Mar 3, 2017
ab8ee6a
Add MANIFEST to .gitignore.
b4hand Mar 3, 2017
459c962
Tell PyPi where README is located.
b4hand Mar 3, 2017
028625e
Merge pull request #63 from seomoz/cbf/ignore-manifest
b4hand Mar 5, 2017
c5113db
Bump rep-cpp version.
b4hand Jun 16, 2017
badeca9
Add str() methods to C++ objects.
b4hand Jun 16, 2017
3771786
Merge pull request #67 from seomoz/bump-rep-cpp
b4hand Jun 19, 2017
5a60622
Release 0.4.7.
b4hand Jun 19, 2017
d6e9841
Merge pull request #68 from seomoz/cbf/release-0.4.7
b4hand Jun 20, 2017
3fffce7
Prefer setuptools to distutils.
b4hand Jun 20, 2017
02e4588
Merge pull request #69 from seomoz/switch-to-setuptools
b4hand Jun 20, 2017
d90ccc6
Add str() method to Agent.
lindseyreno Aug 31, 2017
d3b677f
Merge pull request #71 from seomoz/lindsey/add-str
lindseyreno Aug 31, 2017
f40167e
Fix typo in docstring.
b4hand Sep 6, 2017
93eccb6
Ignore Allow/Disallow directives for external URLs.
b4hand Sep 8, 2017
57cffb7
Merge pull request #72 from seomoz/ignore-external-directives
b4hand Sep 8, 2017
d70138b
Shorten test case names.
b4hand Sep 8, 2017
ed0fc6d
Fix circular dependency on reppy/robots.so
b4hand Sep 8, 2017
a7549f2
Remove `reppy/*.so` during `make clean`.
b4hand Sep 8, 2017
4e91570
Merge pull request #73 from seomoz/makefile-fixes
b4hand Sep 8, 2017
12b45f2
Add fetch hook support.
b4hand Sep 8, 2017
c173a8d
Merge pull request #74 from seomoz/add-fetch-hooks
b4hand Sep 8, 2017
c245fed
Release version 0.4.8.
b4hand Sep 8, 2017
6145fbf
Merge pull request #75 from seomoz/release-v0.4.8
b4hand Sep 11, 2017
67dce37
Add new py3 version to CI (#77)
stevelle Oct 9, 2017
f56a4b7
Fix tox.ini to partially work.
b4hand Oct 23, 2017
bdf492f
Drop tox.ini since it doesn't fully work.
b4hand Oct 23, 2017
ec1c823
Drop unused make target.
b4hand Oct 23, 2017
751e732
Merge pull request #78 from seomoz/cbf/unused-make-target
b4hand Oct 23, 2017
9d8c9cd
Ensure after_response_hook is called on errors.
b4hand Nov 15, 2017
bc65836
Merge pull request #79 from seomoz/call-hook-on-error
b4hand Nov 15, 2017
5c2d2ee
Lower log level for TTL extraction to warn.
b4hand Nov 17, 2017
450a6c3
Merge pull request #81 from seomoz/lower-log-level
b4hand Nov 20, 2017
86eea28
Add Crawl-Delay to string representation of Agent.
b4hand Nov 17, 2017
0247f7b
Fix __str__ on Python 3.
b4hand Nov 20, 2017
29cebd4
Try upgrading Cython.
b4hand Nov 20, 2017
9f42649
Merge pull request #82 from seomoz/crawl-delay-str
b4hand Nov 20, 2017
fcae85a
Update local Python version to 2.7.15.
b4hand May 12, 2018
f899360
Merge pull request #85 from seomoz/update-py2
b4hand May 12, 2018
af22c24
Drop unused imports.
b4hand May 31, 2018
37b52e1
Merge pull request #88 from seomoz/cbf/drop-unused-imports
b4hand May 31, 2018
c1dd314
Python 3.7 is now released. So, test for 3.7.
b4hand Jul 10, 2018
ac3d907
Use `sudo: true` for 3.7 on xenial.
b4hand Jul 10, 2018
a7e196a
Merge pull request #94 from seomoz/python-3.7-travis
b4hand Jul 11, 2018
a95a35b
Bump to latest rep-cpp.
b4hand Jul 23, 2018
cbadb8b
Merge pull request #95 from seomoz/latest-rep-cpp
b4hand Jul 24, 2018
312953f
Add latest Python classifiers.
b4hand Jul 24, 2018
9ba099f
Merge pull request #96 from seomoz/update-setup-py
b4hand Jul 26, 2018
2c58e66
Release 0.4.10
b4hand Jul 26, 2018
67b9feb
Merge pull request #97 from seomoz/release-0.4.10
b4hand Jul 26, 2018
d12a7d5
ReppyException type to wrap requests's ReadTimeout
Sep 18, 2018
ab593ce
regenerate robots.cpp with Cython
Sep 18, 2018
c6d4518
Merge pull request #98 from seomoz/evan/read-timeouts
Sep 18, 2018
7dcec7e
Bump version and update maintainer
Sep 19, 2018
ecf82a8
Merge pull request #99 from seomoz/evan/update-version-and-maintainer
Sep 20, 2018
cdfb058
log reppy errors
Oct 8, 2018
09a7671
Add test for str() UnicodeDecodeError problem
Oct 8, 2018
f055d73
Fix UnicodeDecode/Encode errors for Python 2
Oct 8, 2018
0de7f72
update robots.cpp
Oct 8, 2018
a7763ec
Add a test!
Oct 9, 2018
e651e31
tweak logging message
Oct 9, 2018
ba49132
Merge pull request #100 from seomoz/evan/fix-unicode-decode-encode-er…
Oct 9, 2018
b73f919
Merge pull request #101 from seomoz/evan/log-errors
Oct 9, 2018
5b75e42
Rewrite functionality removed from cachetools 3
Nov 14, 2018
6254349
bump cachetools in requirements.txt
Nov 14, 2018
a90c338
removed rogue comment
Nov 16, 2018
d884572
Much simpler way of doing this.
Nov 16, 2018
c1fd8f1
Merge pull request #103 from seomoz/evan/adapt-to-cachetools-3
Nov 29, 2018
1be2512
release 0.4.12
Nov 29, 2018
4cfa558
Merge pull request #104 from seomoz/release-0.4.12
Dec 21, 2018
1ffab12
Support len(agent)
Mar 12, 2019
bac6b05
Release version 0.4.13
Mar 12, 2019
f22f743
Merge pull request #108 from dlecocq/dan/agent-length
lindseyreno May 28, 2019
2554d8d
Updating rep-cpp submodule
May 28, 2019
722b5dc
fix API to match py2 and py3
wumpus Sep 9, 2019
5a7a3a7
test to catch previous api difference
wumpus Sep 9, 2019
fd60980
apparently I have to check this in when I change robots.pyx
wumpus Sep 9, 2019
e76d41e
Version bump to 0.4.14
Sep 11, 2019
8a5ea6d
Merge pull request #115 from dlecocq/dan/wumpus
lindseyreno Sep 16, 2019
cb57131
Version & Contact Info Update. (#123)
scriptedworld Mar 12, 2020
1de4267
Point to new rep-cpp, which points to new url-cpp, fixing <limits>
tdufala Jan 12, 2024
e92f22e
Merge pull request #133 from seomoz/tdufala/include-limits
tdufala Jan 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .coveragerc
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
[run]
plugins = Cython.Coverage
22 changes: 15 additions & 7 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,9 +1,17 @@
# Binaries and build artifacts
*.pyc
build/*
dist/*
reppy.egg-info/*
*.o
*.so
build/
dist/
reppy.egg-info/
MANIFEST

# Dev artifacts
venv*

# Coverage
.coverage
htmlcov/
.tox
*.egg
*.egg-info

# Vagrant
.vagrant/
3 changes: 3 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[submodule "reppy/rep-cpp"]
path = reppy/rep-cpp
url = https://github.com/seomoz/rep-cpp
1 change: 1 addition & 0 deletions .python-version
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
2.7.15
30 changes: 21 additions & 9 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,14 +1,26 @@
language: python

python:
- "3.4"
- "3.3"
- "2.7"
- "pypy"
matrix:
include:
- python: "2.7"
dist: xenial
sudo: false
- python: "3.3"
dist: trusty
sudo: false
- python: "3.4"
dist: trusty
sudo: false
- python: "3.5"
dist: xenial
sudo: false
- python: "3.6"
dist: xenial
sudo: false
- python: "3.7"
dist: xenial
sudo: true

install:
- if [[ $TRAVIS_PYTHON_VERSION == 2* ]]; then travis_retry pip install -r dev-requirements.txt; fi
- if [[ $TRAVIS_PYTHON_VERSION == 3* ]]; then travis_retry pip install -r dev-requirements-py3.txt; fi
- if [[ $TRAVIS_PYTHON_VERSION == 'pypy' ]]; then travis_retry pip install -r dev-requirements-py3.txt; fi
install: pip install -r dev-requirements.txt

script: make test
7 changes: 7 additions & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
include reppy/rep-cpp/deps/url-cpp/src/*
include reppy/rep-cpp/deps/url-cpp/include/*
include reppy/rep-cpp/src/*
include reppy/rep-cpp/include/*
include reppy/robots.pyx
include reppy/robots.pxd
include reppy/robots.cpp
10 changes: 5 additions & 5 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
.PHONY: test
test:
test: reppy/robots.so
nosetests --with-coverage tests

reppy/%.so: reppy/%.py* reppy/rep-cpp/src/* reppy/rep-cpp/include/* reppy/rep-cpp/deps/url-cpp/include/* reppy/rep-cpp/deps/url-cpp/src/*
python setup.py build_ext --inplace

install:
python setup.py install

dev-requirements:
pip freeze | grep -v -e reppy > dev-requirements.txt

dev-requirements-py3:
pip freeze | grep -v -e reppy > dev-requirements-py3.txt

clean:
rm -rf build dist *.egg-info
rm -rf build dist *.egg-info reppy/*.so
find . -name '*.pyc' | xargs --no-run-if-empty rm
Loading