Node.js support

[dprothero]

It would be interesting to be able to run a search server-side. I know that's not the goal of this project, so this is probably an edge case.

For me, I would like other applications to be able to programmatically invoke my search engine and integrate the results into their own UI (for federated search across multiple sites).

I know one way would be to use a headless browser. I might PoC that, but it doesn't seem like it would be efficient.

I tried importing pagefind.js in a Node script (see below), and got an error SyntaxError: Cannot use 'import.meta' outside a module.

exports.handler = async function (event) {
  const pagefind = await import('../../build/_pagefind/pagefind.js');
  const search = await pagefind.search('kubernetes');

  return {
    statusCode: 200,
    headers: {
      'Content-Type': 'application/json',
    },
    body: JSON.stringify(search),
  };
};

[bglw]

It's a fun angle, and potentially has something to it, but as it exists now it would need some work to run via Node. The import.meta would be resolved by using a type: "module" npm package, but beyond that there will definitely be things that are hit. Pagefind looks at some things on the window object, as well as the URL of the page. It also loads the webassembly via a url, which would need to instead load the file and run through something like https://wasmer.io/ (or have a native build of the code within the WASM)

Definitely non-trivial to add overall, so it isn't something that would come soon from my end. But it's a fun area to discuss so we can keep this convo open — if anyone reading that wants to tackle it, or has any great ideas for making it less work to tackle, jump in! Keen to hear more 🙂

[dscho]

@bglw I would be interested in this, if only to be able to provide better debug logging (and maybe automate tuning ranking parameters via a command-line script). Would you have any concrete pointers, e.g. should this be in a new crate, if yes, should it depend on pagefind_web, or maybe things should be refactored into a common base crate? I will only have time to dabble with this a little bit on the side, but as I said, I'd be very interested.

[bglw]

👋 @dscho

I think a pagefind_search common base crate is the way to go, used as a dep for pagefind_web.

I was originally thinking then that we would then create some new pagefind_something crate for searching, but...

I now think we should give the main pagefind binary the ability to search:

1. The Pagefind searching logic is pretty tight since it needs to compile to wasm, so it's a drop in the bucket for the size of the main binary
2. The main binary already has the NodeJS service logic, so we can just tack onto that
3. Eventually we want to investigate things like incremental builds, which will require that the main binary can load and parse its index from disk, so the functionality aligns somewhat

Hopefully that's enough to go on! 😄 Let me know if you want any pointers to specific files/functions. (also, more than happy to do the last 20% if you block something out in the right direction!)

[bglw]

One extra concrete tip would be to avoid any extra crates in the pagefind_search crate, as it's easy to balloon the wasm build unintentionally there. Importantly, no serde 😅

[dscho]

Speaking of ballooning, it looks as if stems_from_term() relies on the current language being defined as a feature, i.e. it wants to be in a language-specific crate, and this function is called both from exact_term() as well as from search_term(). Meaning: the pagefind_search crate would have to be built per-language. As a consequence, if we did it that way, the pagefind executable would either have to be built once per language or depend on all of those language-specific pagefind_search crates.

The only way out that I can think of is to do away with language-specific features in pagefind_web and instead bundle all of them language-specific stemmers, choosing the right one at runtime. However, that's quite a bit bigger in scope than I was prepared to tackle.

So I went back to the drawing board and thought that maybe I could hack up a node.js script that uses pagefind.js as-is, faking the browser environment that module expects. And that hack actually works! Most of it is concerned with command-line parsing, a smaller part with importing the ESM module from a .js file (i.e. not an .mjs file), a slightly larger part fakes some global objects/functions, and the smallest part is concerned with actually running Pagefind. Here it is, that hack, in its full (current) glory:

(async () => {
  const options = {
    basePath: {
      option: '--base-path',
      value: 'pagefind',
    },
    language: {
      option: '--language',
      value: 'en',
    },
    limit: {
      option: '--limit',
      type: 'number',
      value: 10
    },
    json: {
      option: '--json',
      type: 'boolean',
      value: false
    }
  }
  let terms = []
  
  // Parse command-line options
  const args = process.argv.slice(2)
  optionLoop:
  for (let i = 2; i < process.argv.length; i++) {
    for (const key of Object.keys(options)) {
      const option = options[key]
      if (option.type === 'boolean') {
        if (process.argv[i] === option.option) option.value = true
        else if (process.argv[i] === `--no-${option.option}`) option.value = false
        else continue
      } else if (process.argv[i] === option.option) {
        if (++i >= process.argv.length) throw new Error(`\`${option.option}\` requires an argument`)
        option.value = option.type === 'number' ? Number.parseInt(process.argv[i]) : process.argv[i]
      } else if (process.argv[i].startsWith(`${option.option}=`)) {
        const value = process.argv[i].slice(option.option.length + 1)
        option.value = option.type === 'number' ? Number.parseInt(value) : value
      } else continue
      continue optionLoop
    }
    if (process.argv[i].startsWith('-')) throw new Error(`Unknown option: ${process.argv[i]}`)
    terms.push(...process.argv.slice(i))
    break
  }

  if (terms.length === 0) throw new Error('No search terms provided')

  // Make `basePath` an absolute path
  const path = await import('path')
  if (!path.isAbsolute(options.basePath.value)) options.basePath.value = path.resolve(process.cwd(), options.basePath.value)

  const pagefindPath = path.join(options.basePath.value, 'pagefind.js')
  const url = await import('url')
  const pagefindUrl = url.pathToFileURL(pagefindPath).href

  // Cannot use `await import(...)` because `pagefind.js` does not have the `.mjs` file extension
  // yet is an ESM module. This would elicit the warning and error:
  //
  //   Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension. 
  //   [...]
  //   SyntaxError: Cannot use 'import.meta' outside a module
  //
  // Instead, we load the file contents and dynamically import them via a data URL.
  const fs = await import('fs')
  const contents = await fs.promises.readFile(pagefindPath)
  const moduleUrl = `data:application/javascript;base64,${contents.toString('base64')}`;
  // Dynamically import the module from the blob URL
  const pagefind = await import(moduleUrl);

  Object.assign(globalThis, {
    window: {
      location: {
        origin: ''
      }
    },
    document: {
      querySelector: () => {
        return {
          getAttribute: () => {
            return 'en'
          }
        }
      }
    },
    location: {
      href: pagefindUrl
    },
    fetch: async url => {
      const match = url.match(/(.*)\?.*$/)
      const contents = await fs.promises.readFile(match ? match[1] : url)
      return {
        ...contents,
        arrayBuffer: () => contents,
        json: () => JSON.parse(contents),
      }
    }
  });

  pagefind.options({
    basePath: options.basePath.value
  })
  await pagefind.init(options.language.value)

  for (const term of terms) {
    let results = await pagefind.debouncedSearch(term)
    results = options.limit.value < 1 ? results.results : results.results.slice(0, options.limit.value)
    await Promise.all(results.map(async (item, index) => {
      item.index = index
      item.data = await item.data()
    }))
    console.log(`Results for ${term}:\n${options.json.value
      ? JSON.stringify(results, null, 2)
      : results.map(item => `${item.index + 1}. ${item.data.raw_url} (${item.score})`).join('\n')
    }`)
  }
})().catch(console.log)

[bglw]

Remarkable! Colour me surprised that it's possible this way! 😄

Yes, the language-specific compilation for stemming is something I hadn't clicked onto as a problem — it's done this way to reduce the compiled wasm size by avoiding them all existing at runtime — but the way it is feature flagged should provide an okay-ish ramp to implement the runtime check for the binary version (if we do need to implement it).

If this method is performant enough, and it should be since it's the v8 wasm environment from a browser, then perhaps polishing up this flow makes sense for implementing this support!

I would likely take the angle of avoiding the pagefind.js script altogether, and porting its contents into a node-specific package, since it is currently just a thin-ish wrapper around the wasm anyway.