feat: versioned DatasetStorage for reading/writing datasets across protocol versions

[koenvanderveen]

Summary

Mirrors the syft-job migration/storage architecture (PR #9434) in syft-datasets, adding a versioned read/write layer so datasets can evolve their on-disk format while staying compatible with older peers.

• DatasetStorage — the single choke point for all dataset filesystem IO. Reads select a ProtocolCodec by the on-disk protocol version and upgrade the object to the latest version in memory; writes downgrade to the target protocol and let the codec persist it. Nests a MigrationRegistry, peer_schemas, and codecs.
• v<n> path segment — new datasets can live under public/syft_datasets/v<n>/<name>/ (and private/…). Protocol 0 = the last release (syft-client 0.1.117 / syft-dataset 0.1.20): flat layout, no v<n>, no canonical_name/version fields — read and written back byte-compatibly.
• Versioned models — Dataset / PrivateDatasetConfig are now MigratableObjects under models/<object>/v1.py. Object schemas are unchanged since 0.1.117, so only v1 exists; the protocol bump is purely a path-layout + identity-field change. dataset.py is kept as a re-export shim.
• DatasetRef — path-derived identity (owner, name, protocol_version); the dataset name comes from the path, not the spoofable file.
• Broadcast write — a public dataset is a single copy read by the whole audience (unlike jobs, which target one peer), so the write protocol(s) come from the audience's versions (target_protocol_versions_for_peers), defaulting to the widest-compatible (oldest) protocol when no peers are known. peer_schemas is injectable but not yet populated by syft-client, so today datasets are written in protocol 0 and the v1 layout ships dormant (writing v1 now would hide datasets from 0.1.117 peers).
• SyftDatasetManager routes all reads/writes through DatasetStorage and supports multi-version write (one on-disk copy per protocol version in the audience); get_all dedupes copies across layouts, preferring the newest protocol.
• Release tooling — scripts/export_release_artifact.py + generate_release_fixture.py; the protocol-0 / 0.1.20 artifacts and fixtures are derived as if that release had emitted them.
• Tests mirror syft-job/tests/migrations (unit + p2p), including a byte-exact cross-release round-trip over hand-generated fixtures and a multi-version-write test. CI (unit-tests.yml dataset-tests job) and justfile (test-unit-datasets) wired.

Scope

Package core only, per discussion. syft-client wiring (advertising each peer's dataset protocol version and writing in it at create/share) is a follow-up — syft-client currently exchanges only a single client-version string, not per-package schemas.

Testing

• uv run pytest packages/syft-datasets/tests → 40 passed
• just test-unit-fast → 406 passed
• tests/unit dataset + sync_manager + delete suites → green
• pre-commit run --all-files → clean

[pjwerneck]

A few issues found and commented on.

Overall, the high-level design is in line with #9434, but the fact peer_schemas is still not properly wired is a bit concerning. In production, everything still resolves to v0. I'd be more comfortable if v1 and the whole protocol mechanics was fully integrated and tested end-to-end before wiring it to the rest of syft-client.