Skip to content

Make the docs build resilient: drop blockdiag, pin deps, modernize CI#520

Open
lgirdwood wants to merge 6 commits into
thesofproject:masterfrom
lgirdwood:drop-blockdiag
Open

Make the docs build resilient: drop blockdiag, pin deps, modernize CI#520
lgirdwood wants to merge 6 commits into
thesofproject:masterfrom
lgirdwood:drop-blockdiag

Conversation

@lgirdwood

Copy link
Copy Markdown
Member

What

Make the documentation build resilient and able to build on current Python
(3.11–3.14), instead of being stuck on an old, distro-provided interpreter.

Before this branch the docs would not pip install on a modern Python and the
build emitted ~59 warnings. After it, the docs build cleanly on Python 3.11–3.14
with zero version pins required, and the build is down to 2 pre-existing
warnings (a breathe limitation parsing an anonymous union bind_info in
api/component-api.rst).

Why

The fragility was structural: every painful pin (pillow<10, setuptools<81,
the pkg_resources shim) existed only to keep the orphaned
sphinxcontrib-blockdiag alive — and blockdiag was used for a single diagram.
Meanwhile CI ran on a pinned old image and installed the toolchain from distro
packages, so it never exercised a current Python and the rot went unnoticed.

Changes (one concern per commit)

  1. Drop orphaned blockdiag. Convert the one diagram (mpp_layer EDF
    scheduling) to graphviz — already a dependency that renders every other
    diagram — and remove sphinxcontrib-blockdiag + pillow<10 from both
    requirements files. This removes the entire pillow<10 → setuptools<81 → pkg_resources chain.
  2. Pin the supported build with a lockfile. Add scripts/constraints.txt
    (full transitive tree pinned to validated versions). Supported install is now
    pip install -r requirements.txt -c constraints.txt; requirements-lax.txt
    stays loose for drive-by contributions.
  3. Modernize CI. All jobs on ubuntu-latest; supported build runs a Python
    matrix (3.11, 3.13) via setup-python + the lockfile; lax job switched off
    distro packages. Publishing is gated to one canonical Python so the matrix
    doesn't upload the html artifact twice.
  4. Fix the Dockerfile. It was unbuildable (apt install python3.6 on
    ubuntu-22.04). Rebuilt on ubuntu:24.04, venv install from the lockfile,
    Doxygen built out-of-source where the Makefile expects it.
  5. Catch doc/source drift. Build the Doxygen XML in an explicit CI step so
    the strict (-W) build fails the PR on drift instead of silently dropping API
    sections.
  6. Exclude in-tree virtualenvs from exclude_patterns so a local .venv
    doesn't flood the build with ~50 spurious warnings.

⚠️ Depends on a companion sof change

Commit 5 surfaces drift as an error. The 4 doxygengroup: Cannot find group
warnings (alloc_api, pm_runtime, sof_dma_drivers, sof_dma_copy_func) are
caused by those groups having moved into zephyr/include/ in the sof repo,
which is missing from the Doxygen INPUT. That fix is a separate PR to
thesofproject/sof (adds zephyr/include to doc/sof.doxygen.in).

The strict CI build here will fail until that sof PR is merged, and passes
once it is. The two should land together.

Testing

  • Fresh venv install from the edited requirements.txt pulls no
    pillow/blockdiag/setuptools pins; builds on Python 3.14.
  • Lockfile install reproduces the pinned set exactly.
  • With the companion sof change applied locally, all 4 API groups render and the
    warning count drops from 6 to 2.
  • Verified the venv exclusion ignores a simulated in-tree .venv.

lgirdwood and others added 6 commits June 10, 2026 20:14
blockdiag is orphaned upstream, incompatible with pillow>=10, and uses
APIs removed from modern setuptools (pkg_resources). It forced the
pillow<10 / setuptools<81 pins that block building the docs on current
Python (3.13+), see thesofproject#472.

It was used for a single diagram. Convert that diagram
(mpp_layer EDF scheduling example) to graphviz, which is already a
dependency and renders the rest of the diagrams, then remove the
blockdiag extension and the pillow<10 pin from both requirements files.

With this change the documentation builds on current Python with no
version ceilings.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Liam Girdwood <liam.r.girdwood@intel.com>
The supported requirements.txt used unbounded ">=" specifiers (despite a
comment claiming "==" pinning), so a new Sphinx or breathe release could
break the build with no change to this repo, and two contributors could
get different, unreproducible installs.

Add scripts/constraints.txt, a lockfile pinning the full transitive
dependency tree to exact, validated versions. The supported install now
pairs requirements.txt (the readable top-level list) with this lockfile:

  pip install -r scripts/requirements.txt -c scripts/constraints.txt

Wire it into the supported CI job and the docbuild instructions. The lock
targets a modern Python (Sphinx 9 requires >= 3.11), so the CI job now
uses actions/setup-python instead of the runner's system Python. Fix the
stale requirements.txt header and drop the docbuild note about building
pillow, which no longer applies now that blockdiag/pillow are gone.

requirements-lax.txt stays loose for unpinned drive-by contributions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Liam Girdwood <liam.r.girdwood@intel.com>
The CI ran on the pinned ubuntu-22.04 image and, for the lax job,
installed the doc toolchain from distro packages (python3-sphinx, ...).
That combination only ever exercised an old, distro-provided Python, so
a dependency dropping support for current Python went unnoticed until a
contributor hit it locally.

Modernize the workflow:

- Run every job on ubuntu-latest.
- Build the supported (pinned) docs across a Python matrix (3.11, 3.13)
  with fail-fast disabled, so a pinned dependency that stops working on a
  given interpreter fails CI here instead of months later. Installs use
  setup-python + the requirements.txt/constraints.txt lockfile.
- Publish the deploy artifact from a single canonical Python version
  (publish_python) so the matrix does not upload "html" twice.
- Switch the lax job to setup-python + pip install of the loose
  requirements-lax.txt, dropping the fragile distro-package path. Also
  drop the now-obsolete "-j auto" workaround for Sphinx < 1.7.
- Bump checkout and download-artifact to v4.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Liam Girdwood <liam.r.girdwood@intel.com>
The Dockerfile could not build: it ran "apt-get install python3.6" on
ubuntu-22.04, where that package does not exist. Even past that line it
was broken for the current toolchain -- ubuntu-22.04 ships Python 3.10,
which cannot satisfy the supported requirements (Sphinx 9 needs >= 3.11),
it ignored the constraints.txt lockfile, and it built Doxygen in source
(-B sof/doc) while the sof-docs Makefile looks for the XML under
../sof/build_doxygen, so make html never found the API docs.

Rebuild the image on ubuntu:24.04 (Python 3.12):

- Install the toolchain into a venv (sidesteps PEP 668) from
  requirements.txt + constraints.txt, dropping the redundant distro
  python3-sphinx install.
- Build Doxygen out of source into sof/build_doxygen, matching the
  Makefile's SOF_DOC_BUILD default, so no override is needed in the image.
- Update docker-build.sh to copy that directory back to the host.

Also replace the stale "(bad) default" TODO in the CI workflow with an
accurate note: the Makefile default matches the documented sibling
layout, and CI overrides it only because it checks sof out inside the
sof-docs workspace.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Liam Girdwood <liam.r.girdwood@intel.com>
The supported job builds html with -W, so a doxygengroup referencing a
group that no longer exists in sof turns into an error. Build the Doxygen
XML explicitly in its own step (matching the documented two-step flow)
rather than relying on the Makefile's apidocs target to trigger it as a
side effect of "make html". This makes doc/source drift fail the PR with
a clear, attributable step instead of silently dropping API sections.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Liam Girdwood <liam.r.girdwood@intel.com>
A virtualenv created inside the sof-docs checkout (.venv, venv, env, ...)
is otherwise walked by Sphinx, which then warns about the many .rst files
shipped inside installed packages (docutils, etc.) -- around 50 spurious
"document isn't included in any toctree" warnings that drown out real
ones. Add the common venv directory names to exclude_patterns.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Liam Girdwood <liam.r.girdwood@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant