Reworking Drake's website

[jwnimmer-tri]

We'd like to rework some of the content and style of https://drake.mit.edu/. As part of this work, we'll probably need to tweak our workflow for hosting some static assets. This is a tracking issue to explore that topic.

[jwnimmer-tri]

Updates:

• The new website style was created using Jekyll.
• The redesign ate the whole site, not just the landing page.

For reference, docs.bazel.build also uses Jekyll via Dockerfile; their code is at https://github.com/bazelbuild/bazel/tree/master/scripts/docs.

Drake's python styleguide also uses Jekyll (since it's based on the Google guide which does so), our preview tool for that is https://github.com/RobotLocomotion/styleguide/blob/gh-pages/preview_site_jekyll.py.

I guess I will see how easy or hard adding Jekyll to our Drake docs pipeline looks like. An easy local preview is mostly likely the only real challenge.

FYI the Gemfile in the redesign is just:

source 'https://rubygems.org'

gem 'jekyll', '3.8.5'

group :jekyll_plugins do
  gem 'jekyll-paginate'
  gem 'jekyll-sitemap'
end

[jwnimmer-tri]

My proposal in a quick chat with @jamiesnape today (and afterwards elaborated) was:

(1) demote the website out of bazel build //...; update the documentation instructions to make this clear;

(2) switch the default sense of install_prereqs.sh --with-doc to be opt-in instead of opt-out; update the documentation instructions to make this clear;

(3) add Ruby and Jekyll as a docs-only dependency on all platforms; on Ubuntu, we should try very hard to use the stock apt version of Jekyll, even if it means losing the sitemap for now;

(4) create a (shadow, non-hyperlinked) page or two in parallel with the live site to stand up the tools and deploy pipeline;

(5) migrate the current content to Jekyll by transmogrifying rst to md using the current prose and contractor's stylesheet;

(6) incorporate the additional page flow and copy editing changes from the contractors.

[jamiesnape]

jekyll-sitemap might be an issue on bionic (jekyll-paginate is fine). My reading is that it could be optional for anything but the publication of the site, but you would need to contact the developers.

[jwnimmer-tri]

Yes, I figured we would remove the sitemap entirely until we can change the Documentation builds to Focal.

[jwnimmer-tri]

@jamiesnape I'm going to put (1..4) above on the To Do board at https://github.com/RobotLocomotion/drake/projects/2 for your queue.

[jwnimmer-tri]

Related to (1) -- would be good to keep doc/** text under pre-merge CI. So either add a new Publication (without pushing anywhere) build to pre-merge, or add doc/** to "everything". Probably the former.

[jwnimmer-tri]

Update on (3) -- my new advice is to drop macOS from the platforms we support for local documentation preview. There are very few contributors who do not have access to Ubuntu. For Doxygen, Jekyll, and Sphinx, we should only support Ubuntu from now on.

[jwnimmer-tri]

From #14168 review: it seems like bazel build //doc:serve_sphinx //bindings/pydrake/doc:serve_sphinx might be missing from the CI recipe for Documentation builds. We need to ensure that those program still compile, since the docs instruct developers to run them.

[jamiesnape]

https://github.com/RobotLocomotion/drake-ci/blob/425c8239e235a46498e682f6facbac43b03b0334/driver/configurations/bazel/step-build.cmake#L58 is the place to look.

[jwnimmer-tri]

LGTM

[jwnimmer-tri]

Update on checklist, as I understand it now:

• demote the website out of bazel build //...; update the documentation instructions to make this clear;
• switch the default sense of install_prereqs.sh --with-doc to be opt-in instead of opt-out; update the documentation instructions to make this clear;
• Remove --with-doc-only option from macOS setup scripts. Document in https://drake.mit.edu/documentation_instructions.html that local preview is only for Ubuntu users.
• (3) add Ruby and Jekyll as a docs-only dependency on Ubuntu only; we should try very hard to use the stock apt version of Jekyll, even if it means losing the sitemap for now;
• (4) create a (shadow, non-hyperlinked) page or two in parallel with the live site to stand up the tools and deploy pipeline;
• (5) migrate the current content to Jekyll by:
  • (5.1) Transmogrify rst documentation files to md #14577 transmogrifying rst to md using the current prose
  • (5.2) doc: Should incorporate Jekyll styling for reworked Drake website, handle licensing #14575 Re-constructing contractor's styling for Jekyll
  • (5.3) Fix some MVP set of layout / presentation defects prior to going live (a candidate list to consider is in the immediately next post, below)
  • (5.4) doc: Add consolidated deployment tool #14587 doc: Define groupings to smooth our CI integration #14583 Fix deployment tooling so that Drake git controls the Jekyll vs Sphinx trigger.
  • (5.5) ensure that website works under a separate prefix (e.g. modify _config.yml to use a separate root for this deployment) Edit: Not possible. We'll just have to preview locally.
• (6) incorporate the additional page flow and copy editing changes from the contractors. Edit: Nothing to do here.
• (7) Beta testing Drake website beta testing #14695
• (8) doc: Switch website to use Jekyll #14708 PR the //doc:build swap to move Jekyll to the root and sphinx into a subdir. (Small PR for easy revert.)
• (9) doc: Remove the legacy sphinx website #14758 After a few more days, PR to purge the sphinx files.

[jwnimmer-tri]

As I understand it, now that #14531 + RobotLocomotion/drake-ci#119 have merged, we have a live pipeline where *.md files at https://github.com/RobotLocomotion/drake/tree/master/doc/ are published into https://drake.mit.edu/jekyll/ automatically (with the typical ~30 minutes or so delay, i.e., the same as the currently-live website). So for example:

• https://github.com/RobotLocomotion/drake/blob/master/doc/index.md

is published as

• https://drake.mit.edu/jekyll/

(You can "view source" on the blank page to prove the page is not totally empty.)

[jamiesnape]

Update is nightly, but otherwise yes.

[EricCousineau-TRI]

Updated above bullet-points to add in point (5.2), with more info in #14575

[jwnimmer-tri]

As part of the website transition, for a period of time (once #14612 lands tomorrow) we’re going to have two copies of the website on master (in rst format, and md format). PRs that change the text of the docs will have to keep both in sync.

[EricCousineau-TRI]

@ToffeeAlbina-TRI / Jeremy: Who's on point for Python doc Sphinx conversion re-styling?
I'm going to call "not it" ;)

[jwnimmer-tri]

Not sure exactly what you're asking about. Is it about resurrecting #13983 sans the style-licensing snafu?

My vote would be give up on that and leave the https://drake.mit.edu/pydrake/ style unchanged. To resurrect it, we'd probably want to go back to the contractors to revise the style without including derivative content, but it was such a bear to try to get our patches and macOS unicode font support and etc fixed up the first time, because the contractors don't use change control, that the bang-for-the-buck there doesn't seem worth it to me.

In any case, I'm not currently planning on working on it.

[EricCousineau-TRI]

SGTM.

[jamiesnape]

#13983 would need deconstructing and reconstructing, which is tedious, time consuming, and expensive, and once you have done that you might finally be able to see where the code came from and have to dump it anyway. Same problem as the main site.

[jamiesnape]

BTW The current theme (RTD) was completely arbitrarily chosen by me, so maybe just find an FOSS one that you like, does not clash too badly, and conveys all the information as best as possible.

[jwnimmer-tri]

Yes, that's a good point. If we want to improve pydrake API style, a Drake Developer could just start a new project to find a FOSS style they like better, and get that PR'd. That's more sensible that trying to rework the prior attempt. It's a project management question (@ToffeeAlbina-TRI) whether we task anyone to do that, and on what timeframe.

[jwnimmer-tri]

FYI Here's my command-line to compare the inventory of the two websites, useful for finding pages that are missing or mis-named:

jwnimmer@call-cps:~/jwnimmer-tri/drake$ rm -rf ~/tmp/drake-website && bazel run //doc:gen_sphinx -- --out_dir=/home/jwnimmer/tmp/drake-website/sphinx && bazel run //doc:gen_jekyll -- --out_dir=/home/jwnimmer/tmp/drake-website/jekyll && (cd ~/tmp/drake-website/sphinx && find . -name _images -prune -o -name _static -prune -o -type f | sort) > sphinx.txt && (cd ~/tmp/drake-website/jekyll && find . -name images -prune -o -name assets -prune -o -name third_party -prune -o -type f | sort) > jekyll.txt && diff -u sphinx.txt jekyll.txt

[jwnimmer-tri]

The main work is done. Future improvements are tracked in #14757.