Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add timeout, useragent and proxy support. #7

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Add timeout, useragent and proxy support. #7

wants to merge 3 commits into from

Conversation

researcx
Copy link

No description provided.

@researcx
Copy link
Author

researcx commented Jan 12, 2019 •
edited
Loading

Thanks! I've made the pull request.
As Go doesn't support optional function parameters (golang/go#1675), the timeout, useragent and proxy parameters will become required with this code.
This may require users using goscraper to adjust the goscrape function in their code to this variant, which isn't too ideal.

@researcx
fix: icon links are no only with exact rel="ico"
461f730 
Some pages now have icons in links with rel="shortcut icon" for example
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@researcx @badoux