Switch to use webRef as input

[aeschli]

Webref contains machine readable contents extracted from web browser specifications, including descriptions

https://github.com/w3c/webref
https://www.npmjs.com/package/@webref/css

[captainbrosset]

As discussed offline, I agree that webref is a really great package for this project to start relying on. The MDN website itself now also uses webref instead of the mdn/data repo for things like CSS syntax.

As for descriptions, I tend to think that MDN's descriptions might be better suited for this project. MDN is written for end users, while webref comes from w3c specs which are mostly for browser makers.
MDN's prose is also available under Creative Commons.

Now, I do realize that MDN's descriptions are not readily available as a package. But I think extracting it in an automated way shouldn't be too hard either. I started playing with this idea recently here: https://github.com/captainbrosset/mdn-web-data/ (this is just an exploration, not ready for use).

Maybe @dontcallmedom can chime in on the usefulness of the webref descriptions vs. what's on MDN right now (Dom: for context, the present repo is used to provide intellisense in VS Code for CSS/HTML code when you hover over attributes, properties, values. It's also used by Chrome DevTools to provide tooltips on hover of CSS properties).

[dontcallmedom]

+1 that MDN would be a more developer-friendly source of prose than webref; beyond that, if there is any change needed in webref to make it easier to integrate in VS code, @tidoust and I will be more than happy to look into accommodating them.

[captainbrosset]

Now, I do realize that MDN's descriptions are not readily available as a package. But I think extracting it in an automated way shouldn't be too hard either. I started playing with this idea recently here: https://github.com/captainbrosset/mdn-web-data/ (this is just an exploration, not ready for use).

For info, things are moving in the right direction on this topic. The OWD collective is working right now on making MDN data more easily embeddable by others. You can find more information on their project issue here: openwebdocs/project#174. In particular:

The proposal is that Open Web Docs create and regularly update one or more npm packages containing structured documentation for HTML and CSS. This includes:

• HTML elements
• CSS properties, at-rules, selectors, functions, keywords, types

The work has started, and there's already a few bits and pieces that seem promising:

• https://www.npmjs.com/package/mdn-kumascript is a package that can be used to process the macros found in MDN markdown files.
• https://www.npmjs.com/package/mdn-flavored-markdown is a package that can convert MDN markdown into HTML. MDN markdown has a few additions over vanilla markdown.

I've played a bit with these packages in this experimental repo: https://github.com/captainbrosset/webdocs.

[captainbrosset]

@wbamberg is the one working on the above (at the moment), and can provide more information and offer corrections to on top of my previous message.

Also, Will, feel free to take a look at the contents of the web-data directory, which is where the data is extracted. This might give you ideas for what an actual consumer of the MDN data really needs, which might be good for your project.

[wbamberg]

Now, I do realize that MDN's descriptions are not readily available as a package. But I think extracting it in an automated way shouldn't be too hard either. I started playing with this idea recently here: https://github.com/captainbrosset/mdn-web-data/ (this is just an exploration, not ready for use).

For info, things are moving in the right direction on this topic. The OWD collective is working right now on making MDN data more easily embeddable by others. You can find more information on their project issue here: openwebdocs/project#174. In particular:

The proposal is that Open Web Docs create and regularly update one or more npm packages containing structured documentation for HTML and CSS. This includes:

• HTML elements
• CSS properties, at-rules, selectors, functions, keywords, types

The work has started, and there's already a few bits and pieces that seem promising:

• https://www.npmjs.com/package/mdn-kumascript is a package that can be used to process the macros found in MDN markdown files.

• https://www.npmjs.com/package/mdn-flavored-markdown is a package that can convert MDN markdown into HTML. MDN markdown has a few additions over vanilla markdown.

One thing I'd say about this is that while mdn-flavored-markdown is pretty generic, mdn-kumascript is very "opinionated", meaning the macro processing it does is really targeted at extraction of specific parts of MDN pages, and doesn't give you a like-for-like rendering of MDN pages as a complete blob of HTML. It's really supposed to be used by the web-docs-data package, which builds this JSON extract of bits of MDN, webref, and BCD.

The data currently included in web-docs-data is described in https://www.npmjs.com/package/web-docs-data (currently it is only CSS properties, but we will add other parts of CSS and also HTML elements and attributes), but I can quote it here for convenience:

Currently it contains only CSS properties, accessible under the "properties" key of the "css" object:

{
  "css": {
    "properties": {
      ...
      "margin": {}
      ...
    }
  }
}

Each property object may contain:

• "mdn-page": a string containing the URL of the MDN page for the property
• "browser-compatibility": a string containing the BCD query for the property
• "status": an array of strings, each representing a status flag. The strings can be any of the following:
  • "experimental"
  • "deprecated"
  • "non-standard"
• "summary": a string containing the one- or two-sentence summary for the property, in HTML.
• "interactive-example": a string containing the URL of an interactive example for the property
• "syntax-example": a string containing a series of examples demonstrating the property's syntax
• "constituent-properties": present only if this is a shorthand property. An array of objects, one for each property for which this property is a shorthand. Each object has two properties:
  • "text": the name of the property
  • "target": link to the property's page
• "see-also": an array of objects, each representing a link to a related page. Each object has two properties:
  • "text": the link text
  • "target": the URL of the linked page
• "formal-syntax": syntax for the property and all the types that participate in its definition. An object with the following properties:
  • propertySyntax: a string containing the syntax for the property
  • constituents: an object with one item for each type that participates in the definition of the property's syntax. Each item's key is the name of the type, and the item's value is a string containing the syntax for that type.

All these properties should be treated as optional. (though it would be nice to make better guarantees than this).

Also note that at the moment we're not updating the package on any regular schedule, it is experimental right now.