Statically-typed configuration object (donfig retained as the config reader)

[d-v-b]

Summary

Gives zarr.config a statically-typed, dataclass-backed representation in src/zarr/core/config.py, while keeping donfig as the reader for environment variables and YAML config files. zarr.config now provides precise static types for both attribute access (zarr.config.array.order) and the dotted-string API (zarr.config.get("array.order")), plus runtime validation with typo suggestions — while donfig's config-file locations, parsing, and precedence are preserved by construction.

Motivation: donfig stores config as an untyped nested dict, so every value is typed Any and typos in keys go undetected. This models the config as frozen dataclasses (the typed source of truth) and surfaces the familiar string API through hand-written @overloads that relate each dotted key to its value type. The goal is a solidly-typed config, not removing donfig — the reader has no typing value, so donfig keeps doing it.

Note: an earlier revision of this PR removed donfig and reimplemented its env/YAML ingest. Per review feedback (thanks @maxrjones), that reimplementation is exactly where several regressions originated (config-file locations, etc.), so donfig has been reinstated as the reader. The dataclass remains the typed representation; only the reading layer is donfig's.

Design

• Schema: frozen, slotted dataclasses (ZarrConfig + ArraySettings / AsyncSettings / ThreadingSettings / CodecPipelineSettings). codecs is an open Mapping[str, str] subtree so arbitrary codec names still register at runtime.
• Typed string API: hand-written @overloads map each dotted key ("array.order") to its value type (Literal["C", "F"]), plus a str fallback for codecs.*/unknown keys. A drift-protection test asserts every structured key has a matching overload, so the two can't fall out of sync.
• Subtree access: subtree reads return the typed settings dataclass, which also supports donfig-style item access (config.get("array")["order"]) via a small __getitem__ mixin, alongside attribute access (config.get("array").order).
• async namespace: async is a Python keyword, so the dataclass field is async_ while the serialized key stays "async"; the string API (config.get("async.concurrency")) is unaffected.
• Reader: build_config() uses donfig.Config("zarr") to read ZARR_* env vars and YAML files, then flattens the result and applies it onto the typed defaults (unknown keys warned + skipped). donfig owns discovery/parsing/precedence and its documented file locations.
• State model: a process-global _base snapshot plus a ContextVar overlay. A bare config.set(...) updates only the global base, so a permanent set is visible across threads (last-writer-wins, matching donfig); with config.set(...) is promoted to a context-local overlay on __enter__ and unwound on __exit__, so scoped changes don't leak between threads/tasks or into the permanent base.

Backwards compatibility

The public zarr.config surface is preserved, and donfig-backed reading keeps env/YAML behavior identical. Verified across all in-repo consumers (registry, sync, abc/codec, codec_pipeline, group, abc/store, array_spec, metadata v2/v3):

• config.get("a.b.c"[, default]), subtree config.get("codecs", {}).get(key), and subtree item/attribute access
• config.set({...}) (permanent, cross-thread) and with config.set({...}) (scoped), plus the config.set(key=value) kwargs form
• config.reset(), config.refresh(), config.enable_gpu(), config.defaults
• ZARR_FOO__BAR env-var ingestion and YAML config files (donfig's standard locations: ZARR_CONFIG, ~/.config/zarr, /etc/zarr, sys.prefix/site.PREFIXES)
• the deprecations mechanism (set raises BadConfigError on a removed key; get honors the default — both donfig-faithful)

Intentional, ratified behavior changes

• Unknown keys — "validate at the API, parse external input": programmatic config.set/config.get now raise on an unknown structured key (typo detection — the point of the typed config), while env-var / YAML ingest tolerates unknown keys (skips with a warning) so a stray ZARR_* var or version-skewed config file can't crash import zarr. The open codecs.* namespace still accepts arbitrary keys.
• config.defaults now returns a nested dict (donfig returned a one-element list[dict]).
• Subtree reads return a typed settings object rather than a plain dict; attribute and item access both work, but dict-only operations (.keys(), dict(...)) no longer apply to a subtree.

These are documented in the changed changelog fragment.

Dependency change

• donfig>=0.8 retained (now used only as the env/YAML reader).
• No new runtime dependency (pyyaml arrives transitively via donfig, as before).

Testing

• Full suite green locally (excluding the pre-existing, environment-flaky moto S3-server fixture, which CI covers): 6217 passed, 270 skipped, 4 xfailed; mypy clean (strict) via pre-commit.
• tests/test_config_typed.py: schema/path helpers, env + YAML ingest through donfig (codecs-block deep-merge, directory scanning, precedence, unknown-key tolerance, ~/.config/zarr discovery), the promote-on-enter state model incl. cross-thread visibility, context-local with blocks, the nested-permanent-set no-leak case, subtree item access, deprecations, the overload drift-protection test, and assert_type static-typing assertions.

Notes for reviewers

• The implementation followed an approved design spec and implementation plan, available here as a gist: https://gist.github.com/d-v-b/2a95ff0104824ef52545ed9baf1b66c3

---

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 93.84615% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.50%. Comparing base (1ab9953) to head (ae7d5b3).

Files with missing lines	Patch %	Lines
src/zarr/core/config.py	93.84%	16 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main    #4101    +/-   ##
========================================
  Coverage   93.50%   93.50%            
========================================
  Files          90       90            
  Lines       11981    12228   +247     
========================================
+ Hits        11203    11434   +231     
- Misses        778      794    +16     

Files with missing lines	Coverage Δ
src/zarr/core/config.py	93.96% <93.84%> (-6.04%)	⬇️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[d-v-b]

I'm adding some useful error messages:

  5 Traceback (most recent call last):
  4   File "<stdin>", line 1, in <module>
  3   File "/home/d-v-b/wt/statically-typed-config/src/zarr/core/config.py", line 483, in get
  2     raise _unknown_key_error(key, current) from None
  1 KeyError: "'arraya' is not a valid configuration key. Did you mean 'array'?"

[d-v-b]

flagging this as nice, but delicate (polite term for brittle). These overloads have to be kept synced up with the actual structure of the configuration, which is defined elsewhere. This PR adds tests that confirm that these overloaded signatures match the field name + type of the config object. I'm not sure how we can do do any better than that, if we want a string-based API like this to be statically typed.

an alternative that we can add later would be a non-string-based API, where we define an object that acts like a cursor / lens over our config.

[d-v-b]

donfig returned defaults as a list, we just return a dict

[d-v-b]

I was motivated to open this PR because I think we need a more explicit data structure for our config. Donfig worked well but a morass of untyped dicts is not the UX we want if we plan to be agile with deprecations and feature additions.

[maxrjones]

I really like this direction @d-v-b! Thanks for working on it.

I'll give it a spin, likely on Monday.

[maxrjones]

Thanks for your work on this @d-v-b!

I like that this brings validation of configuration settings along with suggestions for fixing wrong configuration inputs. I'm neutral about donfig vs. just pyyaml as a dependency. I don't think replacing donfig is strictly necessary for getting strict type checking, but also donfig doesn't bring a whole lot of extra value as a dependency such that it's motivating to keep it.

There are some regressions relative to donfig that I think should be addressed or at a minimum included in a changelog fragment under the changed category.

The hybrid state model leads to cross-thread permanent-set shadowing

import threading
from zarr.core.config import ZarrConfigManager
cfg = ZarrConfigManager(); cfg.set({"async.concurrency": 1})
t=threading.Thread(target=lambda: cfg.set({"async.concurrency": 999})); t.start(); t.join()
print("branch main sees:", cfg.get("async.concurrency"))
# -> branch main sees: 1
import threading
from donfig import Config
c = Config("z", defaults=[{"async": {"concurrency": 10}}]); c.set({"async.concurrency": 1})
t=threading.Thread(target=lambda: c.set({"async.concurrency": 999})); t.start(); t.join()
print("donfig main sees:", c.get("async.concurrency"))
# -> donfig main sees: 999

Subtree .get() returns a dataclass

from zarr.core.config import config
print("type:", type(config.get("array")).__name__)
config.get("array")["order"] # used to work

Previous config file locations are no longer picked up

See inlined comment + https://donfig.readthedocs.io/en/latest/configuration.html#yaml-files

[d-v-b]

I like that this brings validation of configuration settings along with suggestions for fixing wrong configuration inputs. I'm neutral about donfig vs. just pyyaml as a dependency. I don't think replacing donfig is strictly necessary for getting strict type checking, but also donfig doesn't bring a whole lot of extra value as a dependency such that it's motivating to keep it.

the core of this PR is defining our config as a dataclass that we manage. how could we do that, and still use donfig? iirc, donfig just exposes an untyped dict-like API. So there would be no way to relate the contents of the donfig config with our normative dataclass. Maybe I'm missing some donfig feature or something?

[maxrjones]

I like that this brings validation of configuration settings along with suggestions for fixing wrong configuration inputs. I'm neutral about donfig vs. just pyyaml as a dependency. I don't think replacing donfig is strictly necessary for getting strict type checking, but also donfig doesn't bring a whole lot of extra value as a dependency such that it's motivating to keep it.

the core of this PR is defining our config as a dataclass that we manage. how could we do that, and still use donfig? iirc, donfig just exposes an untyped dict-like API. So there would be no way to relate the contents of the donfig config with our normative dataclass. Maybe I'm missing some donfig feature or something?

I think if you consider the goal being managing our config internally as dataclasses, donfig must go. If the goal is provide typing and suggestions for configuration settings, that can be built on top of donfig since types can be made more strict.

[d-v-b]

If the goal is provide typing and suggestions for configuration settings, that can be built on top of donfig since types can be made more strict.

Can you outline how this would work?

[d-v-b]

maybe this aligns with your thinking, but I'm going to explore keeping donfig as the config reader, and use the dataclass as the source of truth for the structure of the config