Support blogging-like functionality

[fgregg]

I love where this project is going!

I wonder if y'all would be open to considering adding support for jekyll's blogging pattern.

Namely, that if your markdown file is in a _posts directory, and if its titled like YEAR-MONTH-DAY-title.MARKUP then bloggy things happen:

For jekyll, this means that,

1. The built html is put into a directory structure to get a nice url that looks like DOMAIN/YEAR/MONTH/DAY/title.html
2. the page's date is made available to the layout template, so that the published date can be automatically inserted
3. the site's posts are made available as an object you could interact with sites.posts i.e. to show the latest blog post on the front page jekyll lets you do something like this in an index.md.

---
layout: default
title: My Cool Blog
---

<h1>{{ site.posts.first.title }}</h1>
<p class="text-muted">{{ site.posts.first.date | date: '%B %d, %Y' }} | <a href="{{ site.posts.first.url }}">Permalink</a></p>

{{ site.posts.first.content }}

4. jekyll does other things as well, but those would be the core of it.

I know that blogs are not your first target use case, but I think it could be a great place to go!

I'd be happy to contribute to this.

[Fil]

bloggy things happen? :)

[fgregg]

@Fil, sorry for not being more concrete. I put some details in it.

[Fil]

Interesting.

I don't know how better it is to write 2024-04-01-april-fool.md than having files in folders such as 2024/04/01-april-fool.md. It's probably convenient for some, but is the gain worth the cost, that is, losing the univocal mapping between file and url, and making relative links even more of a headache? Conversely, if you want a flat folder, can't you live with flat urls?

The "page's date is made available to the layout template" is related to #1036. I do want to have meta data (dates, authors, tags…) available to the page and to server-side rendering (SSR, #413).

Inferring a date from a file name seems like a minor affordance; it could be an application of a hook for inferring the page's meta data from front matter & file contents (it's already the case with the title, which can come from either front matter or from the first H1).

For page templates there was a previous attempt (in #253) with discussions and suggestions that I think are still relevant.

It would be interesting to be able to show a (structured, formatted…) list of all pages on a given page, maybe also through SSR — the SSR function could itself have access to the configuration, or even read all the contents on disk (like minisearch does).

[fgregg]

i think the only reason to prefer the jekyll way of handling dates and filenames is that it would make it easier for people who already have jekyll blogs to adopt framework for their data-intensive blogging. jekyll is, by far, the most popular markdown-based blogging tool, as it's built into github pages.

[Fil]

Do you think a script that reads a jekyll repo and "converts" the filenames to framework meta data would help? It could be quick if dates/filenames are the only thing that needs to be adapted. I'm wondering how relative links work, though.

[jaanli]

Yes this would work! Claude or GPT-4 should be able to do it; I just started converting my Jekyll website to framework…

[fgregg]

It would be interesting to be able to show a (structured, formatted…) list of all pages on a given page, maybe also through SSR — the SSR function could itself have access to the configuration, or even read all the contents on disk (like minisearch does).

this "logging" feature is probably the minimal necessary and sufficient requirement to support a blog.

[fgregg]

are you thinking of a script that takes a file named `2024-04-01-april-fools.md` with a frontmatter like: ``` === title: awesome post === ``` to a markdown file named `april-fools.md` with a frontmatter like: ``` === title: awesome post date: 2024-04-01 === ``` ? if, so yes. but the bigger issue is not the input format but what the output looks like like. if before my post was at example.com/2024/04/01/april-fools.html and now it’s at example.com/2024-04-01-april-fools.html or example.com/april-fools.html then that’s going to break links.

[Fil]

If the current URL is example.com/2024/04/01/april-fools.html, the script would have to create docs/2024/04/01/april-fools.md with the date in front matter

[fgregg]

got it! yeah, that could be good. I then wonder how you would then collect the files that should go into the list of posts, but you could do that by looking for some attribute in the front matter.

[jaanli]

This should already be done in the side bar no?

[jamesking]

Long time Jekyll user. I would be very interested in using Framework for blogging and have had the same thought as @fgregg which lead me to this discussion.

I don't necessarily think that Framework needs to copy the same conventions as Jekyll.

From my initial investigation, the elements that are missing from Framework that could allow for the creation of a DataVis blog are:

• A way of marking a subset of the files in the docs/ folder as "posts" to be included in the blog
• A way of listing these posts chronologically to enable the templating of a blog front page
• A way to generate and publish an RSS feed from this list of posts

These elements may well be possible now within the feature-set of Framework today. If anyone has a suggestions for implementing them then I would really appreciate pointers.

[Fil]

The new parametrized page loaders certainly make that possible. To make it work we would have to write a script that lists the pages (both for building and for the sidebar), and a page loader that reads them and converts the relative links to scripts and files. Not unsurmountable, but a bit of effort.

[jamesking]

Thanks - I actually think it might be better for me to simply use a nested structure in the src folder (./2024/09/19/article-title.md) rather than go to the trouble of doing something custom for the sake of mimicking Jekyll.

[jamesking]

Update on progress:

Here is a "Jekyll-like" pattern that is working nicely for me so far.

In my src folder I keep each blog post as an .md file in a date-formatted nested subdirectory like this: ./src/2024/09/19/article-title.md

I have a helper script .helpers/load-posts.js that indexes these posts so that I can import them into a posts.md.js page-loader and display an index of posts at the /posts URL. This uses the same gray-matter dependency that Observable Framework already uses, so I'm not adding any more dependencies.

I also realised that I could import this list of posts into my observable.config.js file so that I can populate the pages configuration automatically without needing to keep the list of posts updated.

Side note: It is really neat that Framework's config is a js file as opposed to json or yaml so tricks like this are possible.

My code is in this gist in case it helps others.

[mcmcclur]

I also realised that I could import this list of posts into my observable.config.js file so that I can populate the pages configuration automatically without needing to keep the list of posts updated.

That's pretty much what I do here: https://mcmcclur.observablehq.cloud/mark-on-framework/

The list of dated posts is generated automatically from the config file. A key difference, though, is that I simply copy the config file over into my src directory so that Javascript in the pages can read it. I was literally just wondering if page loaders have access to path information.

Also, I think it might be worth mentioning that the Framework project linked above embeds GitHub Discussions via Giscus as the commenting system.

Thanks!

[Fil]

thanks for sharing! I think you could simplify the recursive scanning by using import {globSync} from "glob"; (glob is already a dependency of framework).

[Fil]

I've finally managed to support opengraph and twitter cards (#178) in pangea; see for example this page.

Since we need absolute URLs and since it only makes sense for public projects, the trick is to use the github repo as the source for the images, instead of adding the images as assets in the data app.

The code can be seen here: https://github.com/fil/pangea/blob/main/observablehq.config.ts#L97 ;
(commit).

My implementation is incomplete (lacking a few of the usual fields), but they should be pretty simple to add. Not sure how to fill the description… maybe from the front-matter — or we ask claude to summarize the page?

[Fil]

Note to self: an alternative technique is to reference the assets as files in Framework, and compute their hash the exact same way framework does. It goes something like this:

import {createHash}  from "node:crypto";

function head({path, title}) {
  const root = "src";
  const ROOT_URL = "https://observablehq.observablehq.cloud/pangea/";
  let image = "";
  try {
    // computes the same hash as framework 🌶
    const contents = readFileSync(join(root, `thumbnail${path}.png`));
    const key = createHash("sha256").update(contents).digest("hex").slice(0, 8);
    const esc_image_url = JSON.stringify(`${ROOT_URL}_file/thumbnail${path}.${key}.png`);
    image = `<link href="/thumbnail${path}.png">
<meta property="og:image" content=${esc_image_url}>
<meta property="twitter:image" content=${esc_image_url}>
`;
  } catch (error) {}

  return `<meta property="og:title" content=${JSON.stringify(
    title
  )}>\n${image}<link rel="icon" href="data:image/svg+xml,<svg xmlns=%22http://www.w3.org/2000/svg%22 viewBox=%220 0 100 100%22><text y=%22.9em%22 font-size=%2290%22>${EMOJI_FAVICON}</text></svg>">
  `;
}

This adds a link[href] that makes Framework build the asset, and in parallel recomputes the same hash. This approach is a bit complicated, but it could be encapsulated in a specialized function.

[mcmcclur]

I've finally managed to support opengraph and twitter cards (#178) in pangea; see for example this page.

I'm not sure if you noticed when I announced my Framework site on the fourm but, if you check out the link, you should recognize it as opengraph enhanced. I set that up by simply using the fact that the head on the configuration file can be a function depending on the title of the page. Thus, you can place any meta tags that you'd like in there - including opengraph tags.

Having said that, I definitely agree that tools built specifically to support opengraph and SEO more generally would be very welcome!

[Fil]

I wrote a postbuild script to add a simple sitemap to pangea, with the proper headers that reference it.

https://github.com/Fil/pangea/blob/e179717808c66526d5e17a933245f7fe6b91851b/package.json#L7-L8

https://github.com/Fil/pangea/blob/e179717808c66526d5e17a933245f7fe6b91851b/observablehq.config.ts#L101

Making a RSS/Atom file could follow the same approach.

[jamesking]

I see, thanks! I hadn't appreciated that the preview server wasn't loading files from the dist folder.

[jamesking]

OK, so RSS is mostly working.

My last hurdle is related to rendering markdown content of posts into the feed using markdown-it (gist). This was working fine until I added a post with an image. Because the image file gets hashed I need to know what the hashed file location will be when generating the feed.

Is there a Framework function I can call that will render the contents of a post, but with hashed references?

[mbostock]

You could build the site (observable build) and then consume the generated HTML.

[jamesking]

Nice idea, thanks. I'll give it a go.

[jamesking]

Got it working and it is a good solution. Thanks for the suggestion.

[Fil]

Adding a script to adopt a src/public/ folder that gets copied onto the dist/ folder before deploying is as simple as:

  "predeploy": "cp -r src/public/* dist/",
  "deploy": "yarn predeploy && yarn observable deploy"

All of this can be done user-side without too much complexity. All we need now is a template (repo) so that users who don't want to have to think through all of this just have to clone the repo to start their blog.

[Fil]

I had seen the announcement and tested the comments system, but hadn't seen the opengraph tags. Can you share how you obtain the absolute url for og:image?

[mcmcclur]

Well, you know where the file is going to go and you know (mostly) what the name is. For example, if I've got the following file in my source:

src/samples/images/groovy_pic.png

Then I know I'm going to have

https://wncviz.com/_file/samples/images/groovy_pic.xxxxxxxx.png

where xxxxxxxx is the hash for the file. You just need to run

npm run build

to find out the hash. Since an image file has no dependencies, you can be sure that it won't change until you change the file.

Obviously, that's a bit of a pain and it would be preferable to have a dedicated approach to automate this procedure. I do hope you include such a tool in the future but I think it's worth the trouble to add these meta tags via the head function for the time being, if you're trying to build a site that might gain some traction.

[Fil]

OK, I thought you had found a clever workaround :) You have to assume that this file is also statically linked somewhere else in the page, then. Otherwise, you have to add it to the head too, like I did in #1199 (reply in thread).

[mcmcclur]

You have to assume that this file is also statically linked somewhere else in the page

Yes, that's true. At one point, I referenced an image that I didn't want to actually display and then hid it with CSS - just so I could I could have an og tag. A bit of a pain but it worked. :)

[jamesking]

I thought I would post a summary of where I have got to with implementing a blog using Framework. Here is the (very minimal) blog if you would like to take a look. My motivation in choosing Framework is that I eventually want to be able to add posts with interactive explanations and data visualisations (using Observable Plot and D3), and host them on my own domain.

Observable Framework is mainly designed for publishing documentation sites and dashboards, so there are some features missing that other static site generators have (like Jekyll, and Astro) which need to be patched in.

Creating an index of posts.

I have previously written how I implemented this here: #1199 (reply in thread).

Issues:

• In preview, the page that displays an index of posts (a page loader posts.md.js) does not update automatically when I add or edit a new post. I currently workaround this by adding and deleting a few whitespace characters in the posts.md.js to nudge it to rebuild that page.

Adding a feed

I created a data loader feed.atom.js that uses the feed library to build an atom feed. Code here: https://gist.github.com/jamesking/9c06a5196c4c89afe066468282de2b90

Because data loaders publish a hashed file, I also have to edit the build script in package.json to "observable build && cp dist/_file/feed.*.atom dist/feed.atom"

To ensure that Framework builds my feed I also needed to reference the data loader from a page, so I have a hidden page containing a single line: FileAttachment("feed.atom")

Issues:

• It is not possible to view/test the feed when running the Observable preview server. Navigating to /feed.atom returns a 404 because the preview doesn't load files from the /dist folder which the build command builds to.
• I am using markdown-it to render a post's md content into the feed. Currently this breaks images embedded in the file, as my script does not know that the image URL will be hashed. Problem described in my comment above Support blogging-like functionality #1199 (reply in thread).
• When rendering the content of a post into the feed, I currently have a hack to remove the first h1 tag in the article body. Otherwise, the title of the post (as defined by the post's frontmatter title attribute) will be repeated in the article. Some kind of templating system would be really helpful here.

Hosting

I am currently using Netlify to build the site, which "just worked" when I connected the project on Github. I am not sure if this will be as easy if I started to rely on non-js data loaders that require installing or if I start to require network calls to load data when building the site.

Wishlist

Some kind of templating system would be very useful. Different types of pages could be rendered using different templates. Posts could be rendered with a post template that might differ from a page or a gallery template for example.

A way to render files to static locations for specific reasons seems important. Hashing all files feels like it is causing problems, especially for images that might need to be consumed from clients like RSS readers, or Open Graph previews on social media.

[jamesking]

Update on adding a feed:

Thanks to a suggestion from @mbostock I ended up converting my feed.atom.js data loader into a plain node script, build-feed.js (gist). This loops through the rendered posts in the .dist/ folder and builds an RSS feed with the content for each post copied directly from the rendered files.

I run this as a post-build step once the site has been built: observable build && node ./helpers/build-feed.js in package.json.

[jamesking]

Note to self: Here is a useful summary of @Fil's implementation of open graph images. #178 (comment) to work on next.

[jaanli]

I am also using observable framework as a blog - appreciate the inspiration @jamesking ! I spent the weekend migrating examples from earlier this year I made to my personal website/portfolio as I transition out of faculty roles and find a position in industry:

https://jaan.li

Open to any feedback! It's barebones, and I still need to add the og:image functionality from #178 (comment).

We will get there! Thank you so much for enabling this! I'm migrating https://jaan.io over to jaan.li from jekyll to observable framework now, in case anyone has tips/pointers :)