feat(storage): add model file storage and path resolution#6860
Draft
tanishqgandhi1908 wants to merge 21 commits into
Draft
feat(storage): add model file storage and path resolution#6860tanishqgandhi1908 wants to merge 21 commits into
tanishqgandhi1908 wants to merge 21 commits into
Conversation
… tests Review fixes on apache#5911: use dataset owner email in retrieveLatestDatasetVersion, replace brittle Option.get/.head with headOption, strip datasets prefix on selection-modal reopen, clarify FileResolver docs; add FileResolver/DatasetFileNode and frontend path-helper tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The datasets logical-path prefix strips four leading segments (datasets/owner/dataset/version); update the cover-image test's input path to include the prefix so the extracted relative path is the file name, not an empty string. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Now that models are coming as a separate resource/table, the leading resource-type segment is what selects the backing table, so an unprefixed path can no longer be routed unambiguously. Make the "datasets" prefix required instead of a tolerated fallback, and migrate existing data. - FileResolver: a dataset path must start with the "datasets" segment; unprefixed paths are no longer treated as dataset paths. - pytexera DatasetFileDocument: same rule, mirroring the backend. - FileListerSourceOpExec: parse the now-prefixed datasetVersionPath (skip the "datasets" segment); extracted into a testable helper. - Migration (sql/updates/29.sql): prepend "datasets/" to legacy paths stored in workflow.content and workflow_version.content, covering both the fileName (scan sources) and datasetVersionPath (file lister) operator properties. Only values whose first two segments match an existing (user.email, dataset.name) are rewritten, so local paths and URLs are left untouched; email format is irrelevant (owner may be a username without "@"). Uses create_missing=false and is idempotent. - Example workflows: use datasets-prefixed paths. Tests: FileResolverSpec and WorkflowExecutionsResourceSpec updated; test_dataset_file_document.py updated; new FileListerSourceOpExecSpec. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…l-path-prefix # Conflicts: # frontend/src/app/common/type/datasetVersionFileTree.spec.ts
…format python test - dataset-selection-modal.component.spec.ts: expect the datasets-prefixed selectedPath in version (non-file) mode, matching the emitted path. - test_dataset_file_document.py: apply ruff format (wrap an over-length line) so `ruff format --check` passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The cover-image endpoints resolve a dataset path built as
{owner}/{name}/{coverImage}. With the prefix now required by FileResolver,
these must carry the datasets/ segment; add it in the set-cover,
get-cover redirect, and cover-url handlers so cover images resolve again.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…storage # Conflicts: # sql/changelog.xml
…rage # Conflicts: # common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/DocumentFactorySpec.scala
github-actions
Bot
added
feature
engine
ddl-change
Contributor
Automated Reviewer SuggestionsBased on the
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #6860 +/- ##
============================================
+ Coverage 77.30% 82.71% +5.40%
+ Complexity 3524 143 -3381
============================================
Files 1161 483 -678
Lines 45922 25953 -19969
Branches 5101 3090 -2011
============================================
- Hits 35501 21467 -14034
+ Misses 8840 3730 -5110
+ Partials 1581 756 -825
☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 362 | 0.221 | 26,269/39,898/39,898 us | 🔴 +22.0% / 🔴 +140.4% |
| 🔴 | bs=100 sw=10 sl=64 | 778 | 0.475 | 128,420/150,658/150,658 us | 🟢 -15.4% / 🔴 +38.1% |
| ⚪ | bs=1000 sw=10 sl=64 | 904 | 0.552 | 1,098,501/1,192,938/1,192,938 us | ⚪ within ±5% / 🔴 +14.0% |
Baseline details
Latest main f02dd2f from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 362 tuples/sec | 376 tuples/sec | 754.55 tuples/sec | -3.7% | -52.0% |
| bs=10 sw=10 sl=64 | MB/s | 0.221 MB/s | 0.229 MB/s | 0.461 MB/s | -3.5% | -52.0% |
| bs=10 sw=10 sl=64 | p50 | 26,269 us | 21,526 us | 12,816 us | +22.0% | +105.0% |
| bs=10 sw=10 sl=64 | p95 | 39,898 us | 49,086 us | 16,594 us | -18.7% | +140.4% |
| bs=10 sw=10 sl=64 | p99 | 39,898 us | 49,086 us | 19,806 us | -18.7% | +101.4% |
| bs=100 sw=10 sl=64 | throughput | 778 tuples/sec | 817 tuples/sec | 969.38 tuples/sec | -4.8% | -19.7% |
| bs=100 sw=10 sl=64 | MB/s | 0.475 MB/s | 0.499 MB/s | 0.592 MB/s | -4.8% | -19.7% |
| bs=100 sw=10 sl=64 | p50 | 128,420 us | 119,156 us | 103,584 us | +7.8% | +24.0% |
| bs=100 sw=10 sl=64 | p95 | 150,658 us | 178,047 us | 109,097 us | -15.4% | +38.1% |
| bs=100 sw=10 sl=64 | p99 | 150,658 us | 178,047 us | 117,304 us | -15.4% | +28.4% |
| bs=1000 sw=10 sl=64 | throughput | 904 tuples/sec | 907 tuples/sec | 1,004 tuples/sec | -0.3% | -9.9% |
| bs=1000 sw=10 sl=64 | MB/s | 0.552 MB/s | 0.554 MB/s | 0.613 MB/s | -0.4% | -9.9% |
| bs=1000 sw=10 sl=64 | p50 | 1,098,501 us | 1,099,528 us | 1,002,357 us | -0.1% | +9.6% |
| bs=1000 sw=10 sl=64 | p95 | 1,192,938 us | 1,191,126 us | 1,046,463 us | +0.2% | +14.0% |
| bs=1000 sw=10 sl=64 | p99 | 1,192,938 us | 1,191,126 us | 1,073,661 us | +0.2% | +11.1% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,551.75,200,128000,362,0.221,26268.86,39898.41,39898.41
1,100,10,64,20,2569.70,2000,1280000,778,0.475,128420.21,150657.85,150657.85
2,1000,10,64,20,22124.42,20000,12800000,904,0.552,1098500.76,1192937.92,1192937.92
This was referenced Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
Adds ML models as a first-class versioned-file resource — reuses the existing LakeFS + object-storage engine. A model version is stored as one commit in a per-model repository (
model-{mid}), and a logical path/models/<owner>/<name>/<version>/<file>resolves to a stable physical URImodel:///<repositoryName>/<versionHash>/<file>that can be opened to stream the file's bytes.Since most of this mirrors datasets, the shared path parsing, resolution, and document-read logic is factored out and reused. Adds the
modelandmodel_versiontables (the two the resolver consumes). This combines the schema with the logic that uses it, so the schema change is reviewable in context.Any related issues, documentation, discussions?
Closes #6496
Closes #6497
Part of the "Supporting ML models" umbrella #6494. Stacked on #6495 (#6502)
How was this PR tested?
New and existing unit tests (
sbt "WorkflowCore/testOnly *FileResolverSpec *DocumentFactorySpec *DatasetFileDocumentSpec"):model path resolution, required-prefix enforcement, dataset↔model cross-prefix isolation, and
DocumentFactoryrouting themodelscheme to aModelFileDocument. Also verified end-to-end against a live Postgres: a seededmodel/model_versionrow resolves to the expectedmodel:///…URI and routes to aModelFileDocument.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)