How should I configure token in localStorage?

[Ramin-Bateni]

My target website save token key-value in a cookie after login.
Now I want pass the token manually to the crawler to crawl some pages that need authorization.

How should I pass it to the website in my crawler?
Should I create cookie or pass it as header?

• How should I do it with creating cookie?
• How should I do it with adding header?

Currently I have the following code:

const requestQueue = await RequestQueue.open(website.name);

await requestQueue.addRequests([
    {
      url: website.url,
      headers: {
        Accept: 'text/html',
        token: '5f9c88582ce850eff5ae523a6478c08152d0827d',
      },
    },
  ]);

const crawler = new CheerioCrawler({
  requestQueue: requestQueue,
   .....
});

crawler .run();

[B4nan]

This really depends on how the site works, you should obverse and mimic the real requests, check if they have some Authorization header or if they use set-cookie header instead. You can set the cookies via preNavigationHooks:

preNavigationHooks: [
  async ({ session, request }) => {
    // cookies is an array of objects, the same as you can get from the crawler's session via `session.getCookies()`
    session?.setCookies(cookies, request.url);
  },

[Ramin-Bateni]

It seems the website keep the token in localStorage. So I wrote the following code:

preNavigationHooks: [
    async (crawlingContext,) => {

      const { page, request } = crawlingContext;
      const localStorage = await page.evaluate(() =>

        window.localStorage.setItem(
          'token',
          '8582ce8ff5ae52d5f93a64c08152c87858582ce85f93a640e0827d',
        ),
      );

But it return the following error (Access is denied):

ERROR PuppeteerCrawler: Request failed and reached maximum retries. Error: Evaluation failed: DOMException: Failed to read the 'localStorage' property from 'Window': Access is denied for this document.
at pptr://puppeteer_evaluation_script:2:16

How should I resolve it?