BIG-3884 - Use PSL support from url-cpp.

[dlecocq]

This adds support for:

• PSL use in both punycode and unpunycode forms
• an updated copy of the PSL
• a ~9x speed improvement in PSL lookups (over the pure-python PSL implementation)

Relates to #39 and #29

I'd like to figure out a nice way to allow clients to provide their own lists. Any opinions on which is the preferred interface?

# Option 1
import url as URL
URL.use_psl_path('path/to/your/psl')
URL.use_psl_string('your PSL as a string')

URL.parse('http://foo.example.com/').pld

# Option 2
psl = PSL('path/to/your/psl')
psl.getPLD(URL.parse('http://foo.example.com/').hostname)

# Option 3
psl = PSL('path/to/your/psl')
URL.parse('http://foo.example.com/').pldFromPsl(psl)

@b4hand @lindseyreno @neilmb

[neilmb]

I prefer option 1 where we have a (replaceable) Singleton PSL in the Url package. Otherwise I have to think too often about the dependence of parse results on the particular PSL being used. I'd prefer that hidden somewhere under the covers.

[neilmb]

Do we need to keep an empty install_requires list, or lose the while thing until we need it again?

[dlecocq]

We don't need to keep it.

[neilmb]

We are just naming the PSL by date. Do we need to be more specific about its source or particular version on that date (does this thing change more than daily)?

[dlecocq]

To add some substantiation to the claim I made about speed:

import timeit

setup = '''
import url as URL
urls = map(URL.parse, [
    'http://moz.com',
    'http://amazon.co.uk',
    'http://a.b.domain.biz',
    'http://www.test.ac.jp',
    'http://xn--85x722f.xn--55qx5d.cn'
])'''

stmt = '''
for url in urls:
    url.pld
'''

timeit.repeat(stmt, setup, number=10000)

The results:

# On master
[0.1616840362548828, 0.17512893676757812, 0.161513090133667]

# On dan/url-cpp-psl
[0.017058134078979492, 0.017113924026489258, 0.01880812644958496]

[b4hand]

I too vote for Option 1, but I'd rather you pass a PSL object rather than a path to a string so it let's you mock out the PSL lookup more easily. (eg. URL.use_psl(PSL('path_to_psl')))

[b4hand]

It doesn't look like any of your tests check for a non-trivial TLD like .co.uk.

[dlecocq]

I ended up adding support for option 1, though not as Brandon suggested. Currently it accepts a string, and creates the corresponding url-cpp object. Otherwise we'd have to wrap the PSL class and treat the top-level psl as a generic python object. Not a bad thing -- I just didn't want to take the time to do it right now.

[b4hand]

Can we make sure to create a ticket or issue, so that idea doesn't get lost?

[neilmb]

Is the return value of requests.get string-y enough to work with PSL.fromString, or should this use requests.get(...).text?

[dlecocq]

Whoops! I meant to tack content on there.

[dlecocq]

@b4hand -- yes, I'll make a github issue (mostly because I'm not sure where the best place in Jira would be) - #46

[dlecocq]

Sorry, I may have merged this a little prematurely. Let me know if there's any additional feedback, and I'll happily apply it retroactively.