Returning "[]%" for some domains

[dzoba]

➜  crawl git:(master) ✗ crawl -v https://creativelive.com
[]%

I've confirmed it does work on some domains, and doesn't work on others. Above is the output for a domain it breaks with.

[heathdutton]

I've been noticing this too when trying to crawl a site that is hosted locally. Any idea why?

[markuspfeifenberger]

Same for me: https://franks-travelbox.com - I think that crawl is not supporting HTTPS protocol. I wish this would be possible to have HTTPS support in near future ;-)

[heathdutton]

Agreed, it's the https protocol apparently.

[markuspfeifenberger]

Here is the soltution, to be changed in lib/crawler.js line 63:

    if (urlParts.protocol == "https:" || urlParts.protocol == "http:") {

    var protocol = urlParts.protocol.substr(0, urlParts.protocol.length - 1);
    var port = urlParts.port ? urlParts.port : 80,
        siteCrawler = new Crawler(urlParts.hostname, urlParts.path, port);

    // overrule port for https
    if ("https" == protocol && 80 == port) {
      port = 443;
    }

    // configure crawler
    siteCrawler.interval = 10;
    siteCrawler.maxConcurrency = 10;
    siteCrawler.scanSubdomains = true;
    siteCrawler.downloadUnsupported = false;
    // this is needed for https
    siteCrawler.initialProtocol = protocol;
    siteCrawler.initialPort = port;