Skip to content

chore(operators): Add proper descriptions to text source fields#6804

Open
Neilk1021 wants to merge 11 commits into
apache:mainfrom
Neilk1021:opDescs-clean
Open

chore(operators): Add proper descriptions to text source fields#6804
Neilk1021 wants to merge 11 commits into
apache:mainfrom
Neilk1021:opDescs-clean

Conversation

@Neilk1021

@Neilk1021 Neilk1021 commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Added descriptions to fields in TextSourceOpDesc and TextInputSourceOpDesc. Currently these operators suffer heavily from a lack of information.

image

Any related issues, documentation, discussions?

Closes #6802. Related idea in #6798

How was this PR tested?

Deployed changes locally via docker and ./bin/local-dev.sh up

Was this PR authored or co-authored using generative AI tooling?

No.

@github-actions github-actions Bot added frontend Changes related to the frontend GUI common labels Jul 22, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @Yicong-Huang, @ELin2025
    You can notify them by mentioning @Yicong-Huang, @ELin2025 in a comment.

@Neilk1021

Copy link
Copy Markdown
Contributor Author

@mengw15 Could you please look over this when you get the chance?

@codecov-commenter

codecov-commenter commented Jul 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.30%. Comparing base (f02dd2f) to head (bf79bf9).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff            @@
##               main    #6804   +/-   ##
=========================================
  Coverage     77.30%   77.30%           
  Complexity     3524     3524           
=========================================
  Files          1161     1161           
  Lines         45922    45922           
  Branches       5101     5101           
=========================================
  Hits          35501    35501           
  Misses         8840     8840           
  Partials       1581     1581           
Flag Coverage Δ *Carryforward flag
access-control-service 70.00% <ø> (ø) Carriedforward from a351f44
agent-service 76.76% <ø> (ø) Carriedforward from a351f44
amber 69.09% <ø> (ø) Carriedforward from a351f44
computing-unit-managing-service 20.49% <ø> (ø) Carriedforward from a351f44
config-service 66.66% <ø> (ø)
file-service 67.21% <ø> (ø) Carriedforward from a351f44
frontend 82.58% <ø> (ø) Carriedforward from a351f44
notebook-migration-service 78.94% <ø> (ø)
pyamber 92.15% <ø> (ø) Carriedforward from a351f44
workflow-compiling-service 55.14% <ø> (ø) Carriedforward from a351f44

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

✅ No material benchmark regressions detected

🟢 6 better · 🔴 0 worse · ⚪ 9 noise (<±5%) · 0 without baseline

Compared against main f02dd2f benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🟢 bs=10 sw=10 sl=64 577 0.352 16,617/24,204/24,204 us 🟢 -10.8% / 🔴 +45.9%
🟢 bs=100 sw=10 sl=64 1,224 0.747 79,420/105,424/105,424 us 🟢 -20.0% / 🟢 +26.3%
🟢 bs=1000 sw=10 sl=64 1,436 0.876 702,618/722,343/722,343 us 🟢 -9.0% / 🟢 +43.1%
Baseline details

Latest main f02dd2f from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 577 tuples/sec 578 tuples/sec 754.55 tuples/sec -0.2% -23.5%
bs=10 sw=10 sl=64 MB/s 0.352 MB/s 0.353 MB/s 0.461 MB/s -0.3% -23.6%
bs=10 sw=10 sl=64 p50 16,617 us 16,751 us 12,816 us -0.8% +29.7%
bs=10 sw=10 sl=64 p95 24,204 us 27,148 us 16,594 us -10.8% +45.9%
bs=10 sw=10 sl=64 p99 24,204 us 27,148 us 19,806 us -10.8% +22.2%
bs=100 sw=10 sl=64 throughput 1,224 tuples/sec 1,221 tuples/sec 969.38 tuples/sec +0.2% +26.3%
bs=100 sw=10 sl=64 MB/s 0.747 MB/s 0.745 MB/s 0.592 MB/s +0.3% +26.3%
bs=100 sw=10 sl=64 p50 79,420 us 78,865 us 103,584 us +0.7% -23.3%
bs=100 sw=10 sl=64 p95 105,424 us 131,851 us 109,097 us -20.0% -3.4%
bs=100 sw=10 sl=64 p99 105,424 us 131,851 us 117,304 us -20.0% -10.1%
bs=1000 sw=10 sl=64 throughput 1,436 tuples/sec 1,400 tuples/sec 1,004 tuples/sec +2.6% +43.1%
bs=1000 sw=10 sl=64 MB/s 0.876 MB/s 0.854 MB/s 0.613 MB/s +2.6% +43.0%
bs=1000 sw=10 sl=64 p50 702,618 us 717,354 us 1,002,357 us -2.1% -29.9%
bs=1000 sw=10 sl=64 p95 722,343 us 793,729 us 1,046,463 us -9.0% -31.0%
bs=1000 sw=10 sl=64 p99 722,343 us 793,729 us 1,073,661 us -9.0% -32.7%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,346.83,200,128000,577,0.352,16617.12,24204.29,24204.29
1,100,10,64,20,1634.49,2000,1280000,1224,0.747,79420.03,105424.47,105424.47
2,1000,10,64,20,13931.84,20000,12800000,1436,0.876,702617.91,722343.41,722343.41

@mengw15

mengw15 commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Good direction. One consistency note: the existing ~100 @JsonPropertyDescriptions are short declarative phrases ("Data column for box plot"), whereas these are questions / multi-sentence ("What is the name of the column…?"). Worth aligning to the short-phrase style and keeping only the ones that add real info (the per-line vs whole-file note on attributeType is useful; "Attribute Name" probably doesn't need a sentence).

@aglinxinyuan for input.

@aglinxinyuan

Copy link
Copy Markdown
Contributor

Please fix the format of the file.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves operator documentation for the text source operators by adding richer field-level descriptions (via @JsonPropertyDescription) and updating the property editor styling so multi-line descriptions render cleanly in the Angular UI.

Changes:

  • Added @JsonPropertyDescription annotations to TextSourceOpDesc (attribute type/name, limit, offset) and TextInputSourceOpDesc (text input).
  • Tweaked the property editor SCSS to preserve newlines in form “extra” text (.ant-form-item-extra) rendered via nzExtra.
  • Updated some field titles (e.g., “Limit (lines)”, “Offset (lines)”) to be more explicit.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
frontend/src/app/workspace/component/property-editor/operator-property-edit-frame/operator-property-edit-frame.component.scss Preserve newlines and improve spacing for multi-line property descriptions in the form UI.
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/text/TextSourceOpDesc.scala Add field descriptions/titles for shared text source properties (attribute type/name, limit, offset).
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/text/TextInputSourceOpDesc.scala Add a description for the text input field shown in the operator property editor.
Comments suppressed due to low confidence (1)

common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/text/TextSourceOpDesc.scala:66

  • Same as above for Offset: "large binary" is a single-tuple type but the HideAnnotation regex does not hide Offset for it, even though the executor ignores offset/limit when attributeType.isSingle.
  @JsonPropertyDescription("Number of lines to skip from the start before reading. " +
    "(Not used when reading the whole file as one row.)")

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@Yicong-Huang Yicong-Huang changed the title chore(workflow operators): Add proper descriptions to text source fields chore(operators): Add proper descriptions to text source fields Jul 22, 2026
@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Codecov upload failed.

The deferred Codecov Upload workflow (run) failed, so the coverage report for this PR may be missing. This does not block merge.

To recover: re-run the failed jobs in the run above, or manually dispatch the Codecov Upload workflow with
run_id: 30036486463.

@Yicong-Huang Yicong-Huang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with comments on the description itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common frontend Changes related to the frontend GUI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add descriptions to TextInputSource Operator fields.

6 participants