Make the docs build resilient: drop blockdiag, pin deps, modernize CI#520
Open
lgirdwood wants to merge 6 commits into
Open
Make the docs build resilient: drop blockdiag, pin deps, modernize CI#520lgirdwood wants to merge 6 commits into
lgirdwood wants to merge 6 commits into
Conversation
blockdiag is orphaned upstream, incompatible with pillow>=10, and uses APIs removed from modern setuptools (pkg_resources). It forced the pillow<10 / setuptools<81 pins that block building the docs on current Python (3.13+), see thesofproject#472. It was used for a single diagram. Convert that diagram (mpp_layer EDF scheduling example) to graphviz, which is already a dependency and renders the rest of the diagrams, then remove the blockdiag extension and the pillow<10 pin from both requirements files. With this change the documentation builds on current Python with no version ceilings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@intel.com>
The supported requirements.txt used unbounded ">=" specifiers (despite a comment claiming "==" pinning), so a new Sphinx or breathe release could break the build with no change to this repo, and two contributors could get different, unreproducible installs. Add scripts/constraints.txt, a lockfile pinning the full transitive dependency tree to exact, validated versions. The supported install now pairs requirements.txt (the readable top-level list) with this lockfile: pip install -r scripts/requirements.txt -c scripts/constraints.txt Wire it into the supported CI job and the docbuild instructions. The lock targets a modern Python (Sphinx 9 requires >= 3.11), so the CI job now uses actions/setup-python instead of the runner's system Python. Fix the stale requirements.txt header and drop the docbuild note about building pillow, which no longer applies now that blockdiag/pillow are gone. requirements-lax.txt stays loose for unpinned drive-by contributions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@intel.com>
The CI ran on the pinned ubuntu-22.04 image and, for the lax job, installed the doc toolchain from distro packages (python3-sphinx, ...). That combination only ever exercised an old, distro-provided Python, so a dependency dropping support for current Python went unnoticed until a contributor hit it locally. Modernize the workflow: - Run every job on ubuntu-latest. - Build the supported (pinned) docs across a Python matrix (3.11, 3.13) with fail-fast disabled, so a pinned dependency that stops working on a given interpreter fails CI here instead of months later. Installs use setup-python + the requirements.txt/constraints.txt lockfile. - Publish the deploy artifact from a single canonical Python version (publish_python) so the matrix does not upload "html" twice. - Switch the lax job to setup-python + pip install of the loose requirements-lax.txt, dropping the fragile distro-package path. Also drop the now-obsolete "-j auto" workaround for Sphinx < 1.7. - Bump checkout and download-artifact to v4. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@intel.com>
The Dockerfile could not build: it ran "apt-get install python3.6" on ubuntu-22.04, where that package does not exist. Even past that line it was broken for the current toolchain -- ubuntu-22.04 ships Python 3.10, which cannot satisfy the supported requirements (Sphinx 9 needs >= 3.11), it ignored the constraints.txt lockfile, and it built Doxygen in source (-B sof/doc) while the sof-docs Makefile looks for the XML under ../sof/build_doxygen, so make html never found the API docs. Rebuild the image on ubuntu:24.04 (Python 3.12): - Install the toolchain into a venv (sidesteps PEP 668) from requirements.txt + constraints.txt, dropping the redundant distro python3-sphinx install. - Build Doxygen out of source into sof/build_doxygen, matching the Makefile's SOF_DOC_BUILD default, so no override is needed in the image. - Update docker-build.sh to copy that directory back to the host. Also replace the stale "(bad) default" TODO in the CI workflow with an accurate note: the Makefile default matches the documented sibling layout, and CI overrides it only because it checks sof out inside the sof-docs workspace. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@intel.com>
The supported job builds html with -W, so a doxygengroup referencing a group that no longer exists in sof turns into an error. Build the Doxygen XML explicitly in its own step (matching the documented two-step flow) rather than relying on the Makefile's apidocs target to trigger it as a side effect of "make html". This makes doc/source drift fail the PR with a clear, attributable step instead of silently dropping API sections. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@intel.com>
A virtualenv created inside the sof-docs checkout (.venv, venv, env, ...) is otherwise walked by Sphinx, which then warns about the many .rst files shipped inside installed packages (docutils, etc.) -- around 50 spurious "document isn't included in any toctree" warnings that drown out real ones. Add the common venv directory names to exclude_patterns. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Liam Girdwood <liam.r.girdwood@intel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Make the documentation build resilient and able to build on current Python
(3.11–3.14), instead of being stuck on an old, distro-provided interpreter.
Before this branch the docs would not
pip installon a modern Python and thebuild emitted ~59 warnings. After it, the docs build cleanly on Python 3.11–3.14
with zero version pins required, and the build is down to 2 pre-existing
warnings (a breathe limitation parsing an anonymous
union bind_infoinapi/component-api.rst).Why
The fragility was structural: every painful pin (
pillow<10,setuptools<81,the
pkg_resourcesshim) existed only to keep the orphanedsphinxcontrib-blockdiagalive — and blockdiag was used for a single diagram.Meanwhile CI ran on a pinned old image and installed the toolchain from distro
packages, so it never exercised a current Python and the rot went unnoticed.
Changes (one concern per commit)
mpp_layerEDFscheduling) to graphviz — already a dependency that renders every other
diagram — and remove
sphinxcontrib-blockdiag+pillow<10from bothrequirements files. This removes the entire
pillow<10 → setuptools<81 → pkg_resourceschain.scripts/constraints.txt(full transitive tree pinned to validated versions). Supported install is now
pip install -r requirements.txt -c constraints.txt;requirements-lax.txtstays loose for drive-by contributions.
ubuntu-latest; supported build runs a Pythonmatrix (3.11, 3.13) via
setup-python+ the lockfile; lax job switched offdistro packages. Publishing is gated to one canonical Python so the matrix
doesn't upload the
htmlartifact twice.apt install python3.6onubuntu-22.04). Rebuilt on
ubuntu:24.04, venv install from the lockfile,Doxygen built out-of-source where the Makefile expects it.
the strict (
-W) build fails the PR on drift instead of silently dropping APIsections.
exclude_patternsso a local.venvdoesn't flood the build with ~50 spurious warnings.
Commit 5 surfaces drift as an error. The 4
doxygengroup: Cannot find groupwarnings (
alloc_api,pm_runtime,sof_dma_drivers,sof_dma_copy_func) arecaused by those groups having moved into
zephyr/include/in the sof repo,which is missing from the Doxygen
INPUT. That fix is a separate PR tothesofproject/sof(addszephyr/includetodoc/sof.doxygen.in).The strict CI build here will fail until that sof PR is merged, and passes
once it is. The two should land together.
Testing
requirements.txtpulls nopillow/blockdiag/setuptools pins; builds on Python 3.14.
warning count drops from 6 to 2.
.venv.