Quarto render as CI job?

[johnkerl]

HI there -- one of the things we love about Quarto is that docs are things that get built. Very much in the software-dev culture.

For proof-of-concept purposes I'm committing Markdown/YAML mods to a repo, then quarto render and doing git add and git commit of these "build artifacts" as a separate commit. Then serving up doc pages via GitHub Pages.

This makes publishing easy. And it's OK for me as a developer. But for broader uptake, this "committing object files" approach will perhaps not scale as I'd hoped.

Initial feedback I've gotten is that while people will be able to run quarto render locally to preview, they shouldn't have to -- we want to get to a place where people commit Markdown/YAML source to repos, without committing Quarto-generated HTML to those repos.

I'm now doing some research on GitHub Actions, Dockerfiles, Webhooks, etc. etc. and I'm sure I'll come up with a method that does something like:

• Have a Dockerfile with Quarto installed in it (easy)
• Host that image in ECR (easy)
• Grant a CI job / action commit permission on the repo
• Have that CI job run quarto render from the Docker container, and commit HTML "artifacts" to a branch like rendered or somesuch

This will probably all be doable but I thought I'd ask and see if anyone else has already done such a thing & see if there are any lessons learned, best practices, etc.

[cderv]

Regarding Quarto in CI, we have started working on Github Actions related to Quarto in https://github.com/quarto-dev/quarto-actions

For now, we only have one action to easily install Quarto on all available OS worker in a single easy step (name of action and optionally a pin version).

Usually the next step is for someone using this action to run a step with quarto render of a project. And then depending on where the docs are published either to use one of the available actions to publish to github pages or publsih to netlify or any other solution.

I'll be happy to review any contribution there if you have idea for more actions. We can also add some examples of such workflow that I describe as there is none yet.

It is not docker powered but it is working well for Github Action which is the CI you mentioned.

[cderv]

Great. I tried to have a look and opened discussion there: johnkerl/quarto-ci-render-experiment#1
I don't think it is quarto related : installation and rendering works well !

[johnkerl]

Thanks for the advice @cderv!

Agreed, Quarto seems to be doing its job; I'll need to keep debugging to find out why the 404 still at https://johnkerl.github.io/quarto-ci-render-experiment/.

Things going well:

• https://github.com/johnkerl/quarto-ci-render-experiment/blob/main/.github/workflows/quarto-render.yml on branch main
• https://github.com/johnkerl/quarto-ci-render-experiment/tree/gh-pages/docs on branch gh-pages
• (Still 404 at https://johnkerl.github.io/quarto-ci-render-experiment)
• I suspected a race condition between GHP and our Quarto action; have manually re-run each indepedently to no avail

This is a puzzler, still but indeed Quarto is clearly running admirably in CI.

Thanks for getting me off to such a great start!! :D

[johnkerl]

... derp, found it!

:headdesk:

Pre-CI I'd done quarto render on-laptop and committed HTMLs on branch main; post-CI I am (of course) needing to serve up from branch gh-pages.

Now https://johnkerl.github.io/quarto-ci-render-experiment is live -- thank you! :)

[cderv]

Thanks ! I'll remember to add the steps of activating gh-pages (+ links) in the doc we'll do !

[fuhrmanator]

Thanks ! I'll remember to add the steps of activating gh-pages (+ links) in the doc we'll do !

It doesn't yet appear in the doc that I read, so I also realized I had to do this (and I'm happy to get the CD of my site).

However, I am seeing a redundant action runs when I enabled Pages with the gh-pages branch.
It seems I cannot disable "pages-build-deployment" that runs each time I push (makes me feel guilty).
I Googled and found: https://www.reddit.com/r/github/comments/tismvn/comment/i2jzdrg/?utm_source=share&utm_medium=web2x&context=3 but nothing seems to work to disable it.

ETA: this all is apparently "normal" (although I find it confusing to see the separate action). Here's what is written on https://docs.github.com/en/pages/getting-started-with-github-pages/configuring-a-publishing-source-for-your-github-pages-site:

Your GitHub Pages site will always be deployed with a GitHub Actions workflow run, even if you've configured your GitHub Pages site to be built using a different CI tool. Most external CI workflows "deploy" to GitHub Pages by committing the build output to the gh-pages branch of the repository, and typically include a .nojekyll file. When this happens, the GitHub Actions workflow will detect the state that the branch does not need a build step, and will execute only the steps necessary to deploy the site to GitHub Pages servers.

[johnkerl]

For posterity here's what I'm currently using:

$ cat .github/workflows/quarto-render.yml

name: Render and deploy Quarto files
on:
  push:
  pull_request:

jobs:
  quarto-render-and-deploy:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2

    - name: "Install Quarto"
      uses: quarto-dev/quarto-actions/install-quarto@main
      with:
        version: 0.9.141

    - name: "Quarto render"
      shell: bash
      run: |
        quarto render --fail-if-warnings
        # https://github.com/quarto-dev/quarto-cli/issues/493

    - name: "Deploy to gh-pages"
      uses: peaceiris/actions-gh-pages@v3
      if: github.ref == 'refs/heads/main'
      with:
        # This is GitHub Actions magic; no secrets for us to manage; and this works first-time
        # without any extra configs other than visiting Settings -> Pages in your GitHub repo.
        github_token: ${{ secrets.GITHUB_TOKEN }}
        publish_dir: docs
        destination_dir: docs

[cderv]

I have added examples now and will continue to improve. We'll be writing a new publishing section also in the doc with helpful recommendations. Thanks for sharing back your action !

[eitsupi]

FYI, If you want to define a job with a Docker image with some R packages installed, rocker/verse etc. contains a script to install quarto, so we can use something like this (sample)

name: Build and Deploy
on:
  push:
    branches:
      - main
    paths:
      - "*.qmd"
      - "_quarto.yml"
  workflow_dispatch:

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    container:
      image: rocker/verse:latest
    steps:
      - name: Checkout 🛎️
        uses: actions/checkout@v3

      - name: Install quarto
        run: |
          /rocker_scripts/install_quarto.sh
      - name: Build site
        run: |
          quarto render
      - name: Deploy 🚀
        uses: peaceiris/actions-gh-pages@v3
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          publish_dir: ./docs

[eitsupi]

It may be useful to publish a shell script available from the curl command on a website so that we can immediately install the latest version of quarto on any task runner.

[cderv]

In addition to the install-quarto action - yes we should probably do that. I think I can refactor the current action to make scripts available also for using outside of the action like in docker container.

[fuhrmanator]

I know it's not specifically about PDF rendering, but I gave up on docker CI with Texlive.

I got into a catch-22 where the starting pandoc image is set to a specific version of Texlive (say 2021) and the tlmgr installs (eg fonts) I was doing (to keep the image small) want to install the latest libraries (2022). I don't remember the exact details, but unless you save the image of a good docker build (when pandoc is synced with the latest major version of Texlive) it's a headache.

[cderv]

Issue with TeX Live versioning is that every year (around April), there is a new version (2020, 2021, 2022, ...) and the entire CTAN repo is changed - so 2021 can't use 2022 repo. They are freezing the repo. So usually, if you want to use an older TeX Live, you need to configure the use of a specific repo tied to this version. See https://www.tug.org/texlive/acquire.html#past for historic archive mirror.

One solution that could be interesting in some case where size is not an issue is using a full bundle version. Meaning install TeX Live and all packages (or a set of packages) and then do not install anything else (no update or more packages).

With TinyTeX project, we have recently made available a full bundle of TeX Live + all CTAN packages available. See blog post announcement with the links and the how: https://yihui.org/en/2022/05/tinytex-full/
Installing this version means that all the package that could be used will already be available and no mirror is needed, it coud work in offline mode.

Related discussion for TinyTeX : rstudio/tinytex#293

[shug0131]

This was very helpful, but I found with trial and error some corrections. You'd need

permissions:
  contents: write

and

- name: Deploy 🚀
        uses: peaceiris/actions-gh-pages@v4
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          publish_dir: ./_site`

with the publish_dir line being important. THen it works fine.

[jeremiahpslewis]

Here’s a similar setup to the above ideas, with the added advantage that a Docker image is specified in the .devcontainer folder so that you can do your local work (in VSCode) against an identical computational environment and don’t end up spending time debugging something weird happening in Github Actions that is different from your own environment.

https://github.com/jeremiahpslewis/quarto-reproducible-gh-actions

[cderv]

quarto-actions for GHA was updated to v2 with new workflow due to newest quarto feature which is quarto publish command.

Documentation on publishing with Quarto is coming very soon.

For now,

• Look at the quarto-dev/quarto-actions/publish@v2 action : https://github.com/quarto-dev/quarto-actions/tree/main/publish
• you can already find an example workflow in the quarto-action repo: https://github.com/quarto-dev/quarto-actions/blob/main/examples/quarto-publish-example.yml

`quarto publish help`

❯ quarto publish help

  Usage:   quarto publish [provider] [path]
  Version: 99.9.9

  Description:

    Publish a document or project. Available providers include:

     - Netlify (netlify)
     - GitHub Pages (gh-pages)
     - RStudio Connect (connect)

    Accounts are configured interactively during publishing.
    Manage/remove accounts with: quarto publish accounts

  Options:

    -h, --help              - Show this help.
    --id          <id>      - Identifier of content to publish
    --server      <server>  - Server to publish to
    --token       <token>   - Access token for publising provider
    --no-render             - Do not render before publishing.
    --no-prompt             - Do not prompt to confirm publishing destination
    --no-browser            - Do not open a browser to the site after publishing
    --log         <level>   - Path to log file
    --log-level   <level>   - Log level (info, warning, error, critical)
    --log-format  <format>  - Log format (plain, json-stream)
    --quiet                 - Suppress console output.

  Commands:

    help  [command]  - Show this help or the help of a sub-command.

  Examples:

    Publish project (prompt for provider):  quarto publish
    Publish document (prompt for provider): quarto publish document.qmd
    Publish project to Netlify:             quarto publish netlify
    Publish with explicit target:           quarto publish netlify --id DA36416-F950-4647-815C-01A24233E294
    Publish project to GitHub Pages:        quarto publish gh-pages
    Publish project to RStudio Connect:     quarto publish connect
    Publish with explicit credentials:      quarto publish connect --server example.com --token 01A24233E294
    Publish without confirmation prompt:    quarto publish --no-prompt
    Publish without rendering:              quarto publish --no-render
    Publish without opening browser:        quarto publish --no-browser
    Manage/remove publishing accounts:      quarto publish accounts

Supporting publishing to various places directly in quarto publish means that there is no need to rely on external actions / jobs (like a publish to netlify action) and publishing workflow and scripts are more portable in other system that GHA (like gitlab CI).

I'll update when documentation will be out.

[fuhrmanator]

Thanks for these actions!

For a book project, I don't want to render every format on the CD. I think the default publish renders everything? It would be nice to have an option of what to render on publish (although I got it to work with chaining a render action with a publish action using no-render).

[jjallaire]

Yes it does currently render everything (and the quarto publish command doesn't currently support the --to argument). For now I think your workaround is best.

[cderv]

I have update the documentation in quarto-actions READMEs to show that render: false and path could be set on publish action if a render step is done before.