import FileAttachment #325

mbostock · 2023-12-05T21:24:20Z

This allows you to import and use FileAttachment from ES modules. For example:

import {FileAttachment} from "npm:@observablehq/stdlib";

export const data = FileAttachment("data.csv").csv({typed: true});

Under the hood, this works by rewriting the call to FileAttachment to pass in import.meta.url; this allows the FileAttachment constructor to resolve the path to the file relative to the importing module in the same fashion as import.

Since you can use FileAttachment in ES modules now, I’ve also removed the rewriting of fetch; fetch is now unadulterated, and calls to fetch won’t be statically analyzed and turned into file attachments. The motivation here is to encourage people to standardize on FileAttachment since FileAttachment enforces static analysis; unlike fetch, FileAttachment throws a syntax error if you pass arguments that are not statically analyzable. In addition, while we can rewrite direct calls to fetch, we can’t rewrite indirect calls to fetch such as d3.json and d3.csv, so it’s better to be more explicit and limit automatic resolution to FileAttachment.

Fixes #15.

cinxmo · 2023-12-06T02:09:13Z

docs/javascript/files.md

@@ -2,7 +2,13 @@

 TK Should this be called “working with data”?

-You can load files the built-in `FileAttachment` function or the standard [`fetch`](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch) API. We recommend `FileAttachment` because it supports many common data formats, including CSV, TSV, JSON, Apache Arrow, and SQLite. For example, here’s how to load a CSV file:
+You can load files the built-in `FileAttachment` function. This is available by default in Markdown, but you can import it like so:


Suggested change

You can load files the built-in `FileAttachment` function. This is available by default in Markdown, but you can import it like so:

You can load files with the built-in `FileAttachment` function. This is available by default in Markdown, but you can import it like so:

going to continue reviewing tomorrow

Fil · 2023-12-06T07:24:04Z

Nice! Is there any chance that this could also work in data loaders? This way a loader (say, one that does analysis in ts) could use the product of another loader (one that downloads a dataset).

Currently the import fails with ERR_UNSUPPORTED_ESM_URL_SCHEME, which is a generic problem already discussed in #288.

mbostock · 2023-12-06T13:43:45Z

Maybe @Fil. I think I had a Slack thread where I showed how to get the npm: protocol imports working in a data loader. But getting the import working is sort of beside the point — the API for chaining loaders isn’t really about FileAttachment since data loaders can be written in any language. Instead a data loader needs to be able to fetch a file from the preview server (and we need an equivalent server during build). Maybe we set an environment variable which data loaders can read to know the address of the preview server. We’d also need to detect and error on circular dependencies ideally.

Future work though. Is there an issue already for chaining loaders?

Fil · 2023-12-06T14:19:37Z

Now there is one: #332

cinxmo · 2023-12-06T14:57:31Z

src/javascript.ts

@@ -20,9 +18,11 @@ export interface DatabaseReference {
 }

 export interface FileReference {
+  /** The relative path from the source root to the file. */
  name: string;


I think the name is still the relative path from the page to the file?

docs/ ├── earthquakes.csv └── sub └── index.md

and in docs/sub/index.md we have FileAttachment("../earthquakes.csv"), name is "../earthquakes.csv" vs "./earthquakes.csv"

I can double check but there is a difference between Feature.name, FileAttachment.name in generated code, and FileReference.name. It’s all a bit hard to keep straight.

You are right! 👍 Sorry, I was confused. I’ve updated the comment.

cinxmo · 2023-12-06T15:12:21Z

src/javascript.ts

-  const {imports, fetches} = findImports(body, root, sourcePath);
-  const features = findFeatures(body, root, sourcePath, references, input);
+  const references = findReferences(body);
+  findAssignments(body, references, input);


~~we have good test coverage for transpileJavaScript but it would be nice to have specific unit tests for the find<...> functions. I can work that if it's helpful~~ nvm there are comprehensive test cases at the end

cinxmo · 2023-12-06T15:32:03Z

test/javascript/imports-test.ts

@@ -28,16 +28,173 @@ describe("parseLocalImports(root, paths)", () => {
  });
  it("ignores missing files", () => {
    assert.deepStrictEqual(
-      parseLocalImports("test/input/imports", ["static-import.js", "does-not-exist.js"]).imports.sort(compareImport),
+      parseLocalImports("test/input/imports", ["static-import.js", "does-not-exist.js"]).imports.sort(order),
      [
        {name: "bar.js", type: "local"},
        {name: "does-not-exist.js", type: "local"},


I'm confused by this test case - is {name: "does-not-exist.js", type: "local"} supposed to be included?

Yes, we don’t check for existence, just a valid path.

mbostock force-pushed the mbostock/imported-file branch 9 times, most recently from f83cf7f to 87df0cf Compare December 6, 2023 01:20

import FileAttachment

7e8e722

mbostock force-pushed the mbostock/imported-file branch from 87df0cf to 7e8e722 Compare December 6, 2023 01:26

mbostock marked this pull request as ready for review December 6, 2023 01:30

mbostock requested review from Fil, trebor and cinxmo December 6, 2023 01:30

cinxmo reviewed Dec 6, 2023

View reviewed changes

mbostock added 2 commits December 6, 2023 07:06

fix FileReference docs

3dc9f05

Merge branch 'main' into mbostock/imported-file

416b37e

cinxmo reviewed Dec 6, 2023

View reviewed changes

cinxmo approved these changes Dec 6, 2023

View reviewed changes

mbostock mentioned this pull request Dec 6, 2023

Chaining data loaders #332

Open

Merge branch 'main' into mbostock/imported-file

54a33a8

mbostock enabled auto-merge (squash) December 6, 2023 15:57

mbostock merged commit 23cf7a3 into main Dec 6, 2023
1 check passed

mbostock deleted the mbostock/imported-file branch December 6, 2023 15:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

import FileAttachment #325

import FileAttachment #325

mbostock commented Dec 5, 2023 •

edited

Loading

cinxmo Dec 6, 2023 •

edited

Loading

Fil commented Dec 6, 2023

mbostock commented Dec 6, 2023

Fil commented Dec 6, 2023

cinxmo Dec 6, 2023

mbostock Dec 6, 2023

mbostock Dec 6, 2023

cinxmo Dec 6, 2023 •

edited

Loading

cinxmo Dec 6, 2023

mbostock Dec 6, 2023

	You can load files the built-in `FileAttachment` function. This is available by default in Markdown, but you can import it like so:
	You can load files with the built-in `FileAttachment` function. This is available by default in Markdown, but you can import it like so:

import FileAttachment #325

import FileAttachment #325

Conversation

mbostock commented Dec 5, 2023 • edited Loading

cinxmo Dec 6, 2023 • edited Loading

Choose a reason for hiding this comment

Fil commented Dec 6, 2023

mbostock commented Dec 6, 2023

Fil commented Dec 6, 2023

cinxmo Dec 6, 2023

Choose a reason for hiding this comment

mbostock Dec 6, 2023

Choose a reason for hiding this comment

mbostock Dec 6, 2023

Choose a reason for hiding this comment

cinxmo Dec 6, 2023 • edited Loading

Choose a reason for hiding this comment

cinxmo Dec 6, 2023

Choose a reason for hiding this comment

mbostock Dec 6, 2023

Choose a reason for hiding this comment

mbostock commented Dec 5, 2023 •

edited

Loading

cinxmo Dec 6, 2023 •

edited

Loading

cinxmo Dec 6, 2023 •

edited

Loading