URL Should be same protocol as request

[MeshariSA0]

• In self-host we are running without https, however the response Next is always https.

Error Fixed:
HTTPSConnectionPool(host='172.18.0.8', port=3002): Max retries exceeded with url: /v1/crawl/6ebabc4e-6868-43da-a361-d1560dd55ce6?skip=125 (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1006)')))

[nickscamara]

Thanks @MeshariSA0! Will try to get this in soon!

@mogery i believe we had an issue with req.protocol a while ago in prod, where req.protocol would always return http, hence the enforcement. We might wanna double check on why.

[MeshariSA0]

Thanks @MeshariSA0! Will try to get this in soon!

@mogery i believe we had an issue with req.protocol a while ago in prod, where req.protocol would always return http, hence the enforcement. We might wanna double check on why.

This could be due to internal services communicating in HTTP, so the firecrawl-api container receives the request in HTTP protocol, so it always returns HTTP.

[busaud]

Thank you @MeshariSA0 for the solution, we have implemented it in our local deployment until this gets merged.

[rafaelsideguide]

Hey @MeshariSA0 , I merged the PR with main/origin and ran the tests, but it's failing on this test:

  it("should reject malformed URLs starting with 'http://http'", () => {
    expect(() => url.parse("http://http://example.com")).toThrow("Invalid URL. Invalid protocol.");
  });

Could you update it to return this error for invalid protocols?