Replies: 1 comment 1 reply
-
|
Should I post this as an 'Issue'? I'm not sure if it is, or if I'm just missing something! |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Maybe post a note sometwhere if this place isn't monitored |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi,
Just testing the crawler and it seems to work fine on some sites and just gets one page on others.
Example - www.jfmc.co.uk
It only crawls the first page. Even pointing it at the sitemap results in just that page being crawled.
The site pages are all at the root of directories without a .html etc file being specified, eg https://www.jfmc.co.uk/Sitemap , but that doesn't seem to be a problem on other sites I tested.
Nothing in the robots.txt should break it.
Beta Was this translation helpful? Give feedback.
All reactions