infra(docs): containerize and incremental builds

.github/workflows/ci.yml

@@ -325,7 +325,7 @@ jobs:
 
     - name: Test building docs
       run: |
-        cd docs && ./docker.sh
+        cd docs && ./ci.sh
 
   
   test-readme:

.gitignore

@@ -61,6 +61,8 @@ coverage.xml
 
 # Sphinx documentation
 docs/_build/
+docs/doctrees/
+docs/source/demos/
 
 # PyBuilder
 target/

CHANGELOG.md

@@ -21,6 +21,8 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 * Python: Add Python 12 to CI and document support
 * Docs: Udated dependencies - Sphinx 8, Python 12, and various related
 * Docs: Added nbsphinx - hub url grounding, ...
+* Docs: Redo as a docker compose flow with incremental builds (docker, sphinx)
+* Docs: Updated instructions for new flow
 
 ### Fix
 

docs/README.md

@@ -5,54 +5,62 @@ Uses Sphinx to extract .RST from parent .PY and converts into docs/build/ .HTML
 ## Run
 
 ```bash
-docs $ sudo ./docker.sh
+docs $ ./ci.sh
 ```
 
-This is a shim to `build.sh`, which CI also uses
+It is volume-mounted to emit `docs/_build/html/index.html` , as well as epub and pdf versions
 
-It is volume-mounted to emit `docs/build/html/graphistry.html`
+## Architecture
 
-CI will reject documentation warnings and errors
+### Containerized sphinx and nbconvert
 
-## Develop
+ `ci.sh` runs `docker compose build` and then `docker compose run --rm sphinx` to build the docs
 
-Sphinx has to install project dev deps, so to work more quickly:
+  - It also volume mounts demos/ (.ipynb files) into docs/source/demos
 
-1. Start a docker session:
+  - As part of the run, nbsphinx calls nbconvert and then sphinx to convert the .ipynb files into .rst files
 
-Run from `docs/`: 
+    - nbconvert rewrites relative graph.html urls to assume hub.graphistry.com
 
-```
-docker run \
-    --entrypoint=/bin/bash \
-    --rm -it \
-    -e USER_ID=$UID \
-    -e PIP_CACHE_DIR=/cache/pip \
-    -v $(pwd)/..:/doc \
-    -v ~/.cache/pip:/cache/pip \
-    -v ~/.cache/apt:/cache/apt \
-    sphinxdoc/sphinx:8.0.2
-```
+    - We currently use the preexisting .ipynb output, but in the future should reexecute the notebooks to ensure they are up to date
 
-2. Install deps in a sandbox:
+### Caching
 
-```python
-cp -r /doc /pygraphistry \
-&& apt update && apt -o dir::cache=/cache/apt install -y pandoc \
-&& ( cd /pygraphistry && python -m pip install -e .[docs] ) \
-&& (test -d /doc/docs/source/demos || ln -s /doc/demos /doc/docs/source/demos) \
-&& (cd /docs/docs/source/demos && rm -f /doc/docs/source/demos/demos || echo ok)
-```
+The setup aggressively caches for fast edit cycles:
 
-This prevents your host env from getting littered
+  - Standard docker layer caching
 
-3. Edit and build!
+  - Sphinx caching to avoid recompiling unchanged files on reruns
 
-```bash
-cd /doc/docs
-make clean html SPHINXOPTS="-W --keep-going -n"
-```
+    - docs/doctrees for tracking
+    - docs/_build for output
+
+### Error checking
+
+Sphinx in strict mode:
+
+  - Ex: .rst files that aren't included in any TOC
+
+  - Ex: broken links in .ipynb and .rst files
+
+  - Ex: verifies .ipynb follow .rst conventions like first element is `# a title` and only one of them
+
+### Output artifacts
+
+- The output is docs/_build
+
+  - epub, html, and latexpdf
+
+
+## CI
+
+CI runs `ci.sh` and checks for warnings and errors. If there are any, it will fail the build.
+
+## Develop
+
+- Edit the `demo/` notebooks, `graphistry/` Python code, and `docs/source` .rst files
+
+- Rerun `cd docs && ./ci.sh` to see the changes, with the benefit of docker & sphinx incremental builds
 
-This emits `$PWD/docs/build/html/index.html` on your host, which you can open in a browser
+- Check your results in `docs/_build/html/index.html` or the equivalent epub and pdf files
 
-If there are warnings or errors, it will run to completetion, emit them all, and then exit with a failure. CI will reject these bugs, so you need to make sure it works.

docs/ci.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+set -ex
+
+(
+    cd docker \
+    && docker compose build \
+    && docker compose run --rm sphinx
+)

docs/docker/Dockerfile

@@ -0,0 +1,30 @@
+# Use the official Sphinx base image
+FROM sphinxdoc/sphinx:8.0.2
+
+# Set working directory for docs
+WORKDIR /docs
+
+# Step 1: Install necessary system dependencies including Pandoc, with caching for apt-get
+RUN --mount=type=cache,target=/var/cache/apt \
+    --mount=type=cache,target=/var/lib/apt \
+    apt-get update && apt-get install -y pandoc && rm -rf /var/lib/apt/lists/*
+
+# Step 2: Copy project files needed for pip install (setup.py, setup.cfg, versioneer.py, README.md)
+COPY ../../setup.py ../../setup.cfg ../../versioneer.py ../../README.md ./
+
+# Step 3: Install the project dependencies with pip, including docs extras
+RUN python3 -m pip install --no-cache-dir -e .[docs]
+
+# Step 4: Copy the graphistry source files into the container
+COPY ../../graphistry /graphistry
+ENV PYTHONPATH="/graphistry:${PYTHONPATH}"
+
+# Step 5: Copy the build script into the container
+COPY docs/docker/build-docs.sh /build-docs.sh
+RUN chmod +x /build-docs.sh
+
+# Step 6: Set the working directory for Sphinx to the `source/` folder
+WORKDIR /docs/source
+
+# Step 7: Run the script to build the documentation
+CMD ["/build-docs.sh"]

docs/docker/build-docs.sh

@@ -0,0 +1,10 @@
+#!/bin/sh
+
+# Build the HTML documentation incrementally
+sphinx-build -b html -d /docs/doctrees . /docs/_build/html
+
+# Build the EPUB documentation incrementally
+sphinx-build -b epub -d /docs/doctrees . /docs/_build/epub
+
+# Build the PDF documentation incrementally
+sphinx-build -b latex -d /docs/doctrees . /docs/_build/latexpdf

docs/docker/docker-compose.yml

@@ -0,0 +1,15 @@
+# docs/docker/docker-compose.yml
+version: '3'
+services:
+  sphinx:
+    build:
+      context: ../..  # Use project root as context
+      dockerfile: docs/docker/Dockerfile  # Specify the Dockerfile
+    volumes:
+      - ../../docs/source:/docs/source  # Mount the `docs/source` folder to allow live editing
+      - ../../demos:/docs/source/demos  # Mount the `demos` folder for live editing
+      - ../../docs/_build:/docs/_build  # Mount the Sphinx build output to a directory on the host
+      - ../../docs/doctrees:/docs/doctrees  # Mount doctrees to persist build state
+    environment:
+      - USER_ID=${UID}  # Pass user ID for permission handling
+

docs/source/conf.py

@@ -285,6 +285,12 @@
 # This pattern also affects html_static_path and html_extra_path.
 exclude_patterns = [
 
+    '_build',
+     '**/_build/**',
+    'doctrees',
+    '**/doctrees/**',
+    'demos/.ipynb_checkpoints',
+
     # nbsphinx stalls on these
     'demos/ai/Introduction/Ask-HackerNews-Demo.ipynb',
     'demos/ai/OSINT/jack-donations.ipynb',
@@ -364,9 +370,6 @@
     'demos/ai/OSINT/jack-donations.ipynb',
     'demos/ai/OSINT/Chavismo.ipynb',
 
-    # Preserve exclusion of Jupyter Notebook checkpoints
-    'demos/**/.ipynb_checkpoints/**',
-
 ]
 
 pygments_style = "sphinx"