Allow for content loaders to be used on SSR pages

[louisescher]

Summary

Allow for content loaders to fetch collections or external content at runtime.

Background & Motivation

Astro v5 has introduced loaders, which is an amazing system for loading content from content collections or external services at build time. However, a lot of sites have content that changes frequently. Most of the time, this content is hosted externally, so the page doesn't need to rebuild. It would be amazing to have a system that allows for type-safe, cacheable requests in an easy way (that maybe is also available in conjunction with the local API without much configuration)!

I'm a maintainer of StudioCMS, and we'd love to adopt / create a content loader to be used for the CMS, but the limitations for build time only are a bit problematic for people whose content changes often. I think that there's a lot of content sources here that might benefit from runtime loaders.

Goals

• Modify the content loaders API to allow for fetching (and caching) (external) content at runtime
• If there is no feasible way of modifying the current loaders, provide an alternative for runtime fetching.
• Allow for type-safe requests that can be cached easily.

Example

You could define a loader like usual:

const countries = defineCollection({
  loader: async () => {
    const response = await fetch("https://example.com");
    const data = await response.json();
    // Must return an array of entries with an id property
    // or an object with IDs as keys and entries as values
    return data.map((country) => ({
      id: country.cca3,
      ...country,
    }));
  },
  schema: /* ... */
});

Then, fetch the content with getCollection in the file:

import { getCollection, getEntry } from 'astro:content';

// Get all entries from a collection.
// Requires the name of the collection as an argument.
const allCountries = await getCollection('countries');

Perhaps the behavior could be based on the export const prerender = false | true directive!

[florian-lefebvre]

What would be the advantage of a content loader VS just a SDK in this case? Eg. for directus using getCollection('posts') VS await client.request(readItems('posts'));, it seems easier to use the SDK

[louisescher]

It might be useful for cases where there is no SDK. For example, the podcast index, which only has a REST API. I'm sure there's lots more APIs and libraries where no SDK is available.
Then there's the case of abstracting away some of the complexity with mapping that data, and lastly the caching part, which would be amazing if it'd be handled by Astro / your Adapter.

[Adammatthiesen]

From a StudioCMS standpoint, it would save us from having an entire "SDK" for our system, currently we have a hand crafted set of helpers (that we serve through virtual modules) to be able to grab from the DB, if we had access to the new content Layer, we could provide functions to allow our users (And potentially use internally) an easier API that comes straight from Astro for grabbing the content live during PROD. (Since we are not only a CMS, but have the option to go to a headless mode)

[ematipico]

I am somewhat concerned by the proposal without expected boundaries.

One of the pain points is MDX. From a content layer point of view, MDX can be rendered only at runtime, so you won't ever see the compiled HTML of a MDX file inside the data store of the content layer.

If you allow the content layer to be available in SSR, this means that data coming in a MDX shape needs to be compiled and transformed during the rendering phase. This will make pages very very slow, and we don't want that.

So for this proposal to happen, we need to set proper expectations and we should avoid performance pitfalls.

[louisescher]

Okay, I think some clarification is in order. This proposal imagines runtime content loaders in a way where they are only meant to render remote resources. Local Markdown / MDX files (and other resources, should the prerender flag be set to true) are compiled at build time. This is mainly aimed at providing a way to "write" SDKs for, say, REST-based APIs and services.
In cases where remote content is fetched that has to go through a render process (Markdown, MDX, Markdown etc.), that will of course be a notable performance hit. But it isn't any different with how things work now in an SSR context, if I manually fetch some MDX in the frontmatter of a page I still have to render that anyway.
See https://docs.astro.build/en/guides/markdown-content/#fetching-remote-markdown for more info on that!

[louisescher]

Also, another thing I just remembered. A lot of services (for example, Ghost CMS and Payload CMS) already provide you with pre-rendered HTML strings based on your markdown!

[minht11]

We using Strapi which is does not have SDK and only a rest API. Most of our pages are static but some require SSR. Though all of them have preview functionality, to view how site would look like before redeploy. That requires SSR.

After I upgrading to Astro 5 I had hoped to migrate to the new Content Lake API, but lack of SSR support makes it not possible, better integration with rest api, easier dynamic requests caching, would be greatly appreciated.

[andrioid]

Before the content-loader API, I'd say sure, using an SDK is probably best. But the power it gives site-owners to pop a loader in there and forget about it extremely powerful. Especially because the loader essentially only needs a way to list/index and get individual entries.

With such a lean API, caching and search indexing could also be introduced without burdening the end-user too much.

In comparison, if I wanted to implement content from a CMS SDK (or manually fetching), I'd have to do all the caching and indexing myself without the niceties of having a content-schema defined for my site.

My use-cases: I have bluesky posts, github projects and blog posts from my Obsidian directory (on Nextcloud) that I'd love to have added without having to deploy my SSR site.

This isn't a must-have feature for me, but reading through the API and docs, I assumed that SSR pages would call getEntry and the loader would get the fresh content and periodically index the data source.

While searching for this, I've found many confused people posting issues and questions, so if the plan isn't to enable SSR for loaders, then the docs should probably state that more clearly.