Merge with tosback-crawler

[michielbdejong]

No description provided.

[michielbdejong]

Hi ToS;DR crawler, meet our Tosback crawler :) tosdr/tosback-crawler#8

[michielbdejong]

@JustinBack hi! I think the last time we discussed this, we noted the two crawlers solve different parts of the crawling problem. Can we merge them together somehow?

• crawler.tosdr.org is a proxy that simplifies any webpage using server-side rendering, and then outputs html
• tosback-crawler has rules to know which URLs to crawl and outputs markdown
• edit.tosdr.org has the same rules as tosback-crawler, but outputs its own simplified html syntax

Currently, we are using edit.tosdr.org in front of crawler.tosdr.org and that works well, but there are a number of advantages we can get from tosback-crawler:

• the html parser in tosback-crawler is more powerful than the one in edit.tosdr.org so we could get rid of crawler.tosdr.org. Or in other words, the function of crawler.tosdr.org (server-side renderer, headless browser, html parser, whatever we call it) would be merged into tosback-crawler.
• we would switch edit.tosdr.org over from "funky semi-html with ugly tags" to markdown. That would make it much easier for the users of edit.tosdr.org to review documents, because it's clearer where the headers are, where the bullet points are, etc.

[JustinBack]

It should be possible to merge them together, I'd just have to get into the code tosback-crawler uses.

Technically its possible to switch crawler.tosdr.org over to markdown as well, maybe for testing? I can migrate a specific crawler to makrdown to see how it goes, then merge it into tosback