fix(operator): add 100MB memory limit to binary file scan to prevent OOM

[roshiiiz]

What changes were proposed in this PR?

This PR introduces a strict 100 MB memory allocation limit when scanning files using the standard binary attribute type.

Why is it needed?
Previously, when users mistakenly attempted to read massive files (e.g., >8GB) using the binary attribute type (instead of the streaming large binary type), the ByteArrayOutputStream would attempt to allocate the entire file into a single contiguous Java byte array. Since standard byte[] arrays max out at ~2.14GB, this inherently fails. Worse, on standard developer laptops or lower-memory deployments, this massive allocation attempt triggered immediate IllegalArgumentExceptions or severe JVM Garbage Collection death spirals, causing the computing-unit-master to lock up or crash entirely without reporting a user-friendly error to the frontend UI.

What was changed:

• Added a MAX_SAFE_SIZE limit of 100 MB to FileScanUtils.scala.
• During the binary scan loop, if the byte stream exceeds 100 MB, it is immediately intercepted and aborted safely.
• Throws a clean, user-friendly RuntimeException directly to the frontend directing the user to use the large binary attribute type instead for massive files, completely preventing the JVM from thrashing or freezing.

Any related issues, documentation, discussions?

Closes #3721

How was this PR tested?

Manual Verification:

1. Uploaded an 8.9 GB CSV test file to the workspace.
2. Created a workflow with the File Scan operator.
3. Configured the operator to use the binary attribute type (which intentionally attempts to load the entire file into memory).
4. Ran the workflow.
   Result: The workflow aborted instantly (in ~3-4 seconds) the moment it crossed the 100MB threshold. It safely threw the custom user-friendly error message in the UI and immediately cleaned up the worker thread without any memory leaks or GC pauses.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Antigravity (DeepMind)

Changed it to 100 Mb instead of 1gb afterwards

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• Contributors with relevant context: @aglinxinyuan
  You can notify them by mentioning @aglinxinyuan in a comment.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 55.55556% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.54%. Comparing base (87ec5f2) to head (509246d).
⚠️ Report is 27 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...mber/operator/source/scan/file/FileScanUtils.scala	55.55%	7 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##               main    #6811   +/-   ##
=========================================
  Coverage     75.54%   75.54%           
- Complexity     3452     3456    +4     
=========================================
  Files          1161     1161           
  Lines         45921    45939   +18     
  Branches       5101     5105    +4     
=========================================
+ Hits          34690    34706   +16     
- Misses         9613     9619    +6     
+ Partials       1618     1614    -4     

Flag	Coverage Δ		*Carryforward flag
access-control-service	70.00% <ø> (ø)
agent-service	76.76% <ø> (ø)		Carriedforward from 87ec5f2
amber	67.34% <55.55%> (+0.02%)	⬆️
computing-unit-managing-service	20.49% <ø> (ø)
config-service	66.66% <ø> (ø)
file-service	67.21% <ø> (ø)
frontend	80.04% <ø> (ø)		Carriedforward from 87ec5f2
notebook-migration-service	78.94% <ø> (ø)
pyamber	92.15% <ø> (ø)		Carriedforward from 87ec5f2
workflow-compiling-service	55.14% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 2 worse · ⚪ 11 noise (<±5%) · 0 without baseline

Compared against main 87ec5f2 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
⚪	bs=10 sw=10 sl=64	364	0.222	26,507/36,186/36,186 us	⚪ within ±5% / 🔴 +126.0%
🔴	bs=100 sw=10 sl=64	782	0.477	126,055/158,691/158,691 us	🔴 +14.6% / 🔴 +49.1%
🟢	bs=1000 sw=10 sl=64	894	0.546	1,121,037/1,149,151/1,149,151 us	🟢 -5.8% / 🔴 +15.2%

Baseline details

Latest main 87ec5f2 from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	364 tuples/sec	378 tuples/sec	767.66 tuples/sec	-3.7%	-52.6%
bs=10 sw=10 sl=64	MB/s	0.222 MB/s	0.231 MB/s	0.469 MB/s	-3.9%	-52.6%
bs=10 sw=10 sl=64	p50	26,507 us	25,916 us	12,623 us	+2.3%	+110.0%
bs=10 sw=10 sl=64	p95	36,186 us	36,746 us	16,011 us	-1.5%	+126.0%
bs=10 sw=10 sl=64	p99	36,186 us	36,746 us	18,856 us	-1.5%	+91.9%
bs=100 sw=10 sl=64	throughput	782 tuples/sec	819 tuples/sec	998.44 tuples/sec	-4.5%	-21.7%
bs=100 sw=10 sl=64	MB/s	0.477 MB/s	0.5 MB/s	0.609 MB/s	-4.6%	-21.7%
bs=100 sw=10 sl=64	p50	126,055 us	122,249 us	100,463 us	+3.1%	+25.5%
bs=100 sw=10 sl=64	p95	158,691 us	138,518 us	106,400 us	+14.6%	+49.1%
bs=100 sw=10 sl=64	p99	158,691 us	138,518 us	117,712 us	+14.6%	+34.8%
bs=1000 sw=10 sl=64	throughput	894 tuples/sec	892 tuples/sec	1,034 tuples/sec	+0.2%	-13.6%
bs=1000 sw=10 sl=64	MB/s	0.546 MB/s	0.544 MB/s	0.631 MB/s	+0.4%	-13.5%
bs=1000 sw=10 sl=64	p50	1,121,037 us	1,115,133 us	973,294 us	+0.5%	+15.2%
bs=1000 sw=10 sl=64	p95	1,149,151 us	1,219,263 us	1,019,213 us	-5.8%	+12.7%
bs=1000 sw=10 sl=64	p99	1,149,151 us	1,219,263 us	1,049,896 us	-5.8%	+9.5%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,549.37,200,128000,364,0.222,26507.20,36186.03,36186.03
1,100,10,64,20,2557.87,2000,1280000,782,0.477,126054.83,158690.91,158690.91
2,1000,10,64,20,22361.44,20000,12800000,894,0.546,1121037.36,1149150.74,1149150.74

[aglinxinyuan]

If the Java limit is about 2 GB, should we set it to 2 GB instead of 100 MB? What if I want to upload a 200MB as binary?

[Copilot]

Pull request overview

This PR hardens the File Scan operator’s in-memory (non-streaming) read path by adding a size guard when converting an InputStream into an in-memory byte[], preventing extremely large inputs from triggering JVM OOM/GC thrash and surfacing a user-facing error directing users to the streaming (large binary) option.

Changes:

• Replaces IOUtils.toByteArray usage with a custom safeToByteArray that enforces a 100MB maximum read size.
• Throws a user-oriented exception once the in-memory read exceeds the safe threshold (with guidance based on attribute type).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[roshiiiz]

If the Java limit is about 2 GB, should we set it to 2 GB instead of 100 MB? What if I want to upload a 200MB as binary?

Great question! While the theoretical Java byte limit is 2GB, ByteArrayOutputStream doubles its internal buffer capacity every time it resizes. This means reading a 1GB file actually requires >3GB of free heap space just to perform the array copy. On standard local deployments, this instantly triggers OutOfMemoryErrors or severe GC death spirals that crash the worker JVM before it ever hits the 2GB limit.

The 100MB limit was chosen as a conservative safety net to guarantee the worker never crashes, while actively nudging users to select the large binary attribute type instead for bigger files (since large binary streams the data chunk-by-chunk and creates zero heap pressure).

However, I am completely open to bumping this limit to 500MB if you feel that's a better threshold for standard use cases! Let me know what limit you'd prefer and I'll gladly update the PR.

[kunwp1]

Do you think producing OOM is a bug? I think it's natural to produce OOM error if the operator doesn't support reading more than 2GB file. I am okay if the direction is to make the error message user-friendly and guide to using large-binary type but I don't think introducing a threshold is a good solution.

[roshiiiz]

Do you think producing OOM is a bug? I think it's natural to produce OOM error if the operator doesn't support reading more than 2GB file. I am okay if the direction is to make the error message user-friendly and guide to using large-binary type but I don't think introducing a threshold is a good solution.

I completely understand your point, and I agree that an OOM is natural when hitting that 2GB limit.
The main reason I initially leaned toward a threshold was actually to solve a user experience issue. The original use case that sparked this involved someone uploading a 9GB file. They ended up waiting a really long time for it to process, only for the whole thing to crash at the very end with a generic error. My goal was to introduce a way to 'fail fast' so a new user unaware of the limit doesn't waste 30 minutes waiting on a 10GB file just to get an OOM.

[kunwp1]

If you want a faster fail path, then I wonder why we have to run the operator to check the file size. Did you consider a way to check the file size before running the operator? Also, exceeding 100MB doesn't necessarily mean the operator will cause an OOM. Can you reconsider the idea?

[roshiiiz]

If you want a faster fail path, then I wonder why we have to run the operator to check the file size. Did you consider a way to check the file size before running the operator? Also, exceeding 100MB doesn't necessarily mean the operator will cause an OOM. Can you reconsider the idea?

That is a great suggestion, but checking the file size upfront introduces some edge cases. For instance, if a user uploads a .zip file, we cannot determine the uncompressed size of the extracted entries until the operator is actively streaming and unzipping them on the fly. (I might be overthinking this as well)

However, I completely agree that your original suggestion is 100% the cleanest and best approach. I will go ahead and remove the threshold entirely. I'll update the PR to let it hit the Java array limit, and we can simply catch the resulting OutOfMemoryError or IllegalArgumentException and wrap it in user-friendly large-binary hint.

Thanks for the feedback I will update the PR with this approach shortly.

[kunwp1]

Sounds good! I haven't thought about a zip file but your opinion makes sense that it's not possible to cover all the cases and check the file size beforehand. You can go ahead with the fix.

[aglinxinyuan]

If the Java limit is about 2 GB, should we set it to 2 GB instead of 100 MB? What if I want to upload a 200MB as binary?

Great question! While the theoretical Java byte limit is 2GB, ByteArrayOutputStream doubles its internal buffer capacity every time it resizes. This means reading a 1GB file actually requires >3GB of free heap space just to perform the array copy. On standard local deployments, this instantly triggers OutOfMemoryErrors or severe GC death spirals that crash the worker JVM before it ever hits the 2GB limit.

The 100MB limit was chosen as a conservative safety net to guarantee the worker never crashes, while actively nudging users to select the large binary attribute type instead for bigger files (since large binary streams the data chunk-by-chunk and creates zero heap pressure).

However, I am completely open to bumping this limit to 500MB if you feel that's a better threshold for standard use cases! Let me know what limit you'd prefer and I'll gladly update the PR.

Do you have some data to support this? For example, can you test the maximum possible file size for this operator? We want to set this limit to upper bound instead of lower bound.