fix: read-idle timeout on SSE so a stalled stream can't hang events()#12
Conversation
The live-events SSE reader used timeout=None, so a connection that goes silent-but-open (no frames, no keepalive, no close — e.g. a gateway idle timeout on a long job) blocked iter_lines() forever. The high-level events() reconnect/poll-fallback only runs when the stream *ends*, so it never recovered — events() hung indefinitely on jobs longer than a few minutes, even though the job itself completed fine (poll/wait/run stayed correct). Give get_job_events (sync + async) a read-idle timeout well above the server's keepalive interval: a stalled stream now raises ReadTimeout, which events() already catches and turns into a poll/reconnect. Adds a threaded stub "stall" mode + a regression test proving recovery instead of a hang. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Warning Review limit reached
Next review available in: 48 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
Warning
|
| Layer / File(s) | Summary |
|---|---|
SSE timeout integration src/comfy_low/transport.py |
Sync and async get_job_events methods now apply _SSE_IDLE_TIMEOUT by default, while timeout=None remains an explicit opt-out. |
Stalled SSE fixture tests/conftest.py |
The threaded test server supports stalled SSE connections that emit non-terminal frames and delay before returning. |
Recovery regression test tests/test_sse_idle.py |
The test verifies timeout-driven recovery, progress delivery, successful terminal status, and bounded completion time. |
🚥 Pre-merge checks | ✅ 2
✅ Passed checks (2 passed)
| Check name | Status | Explanation |
|---|---|---|
| Linked Issues check | ✅ Passed | Check skipped because no linked issues were found for this pull request. |
| Out of Scope Changes check | ✅ Passed | Check skipped because no linked issues were found for this pull request. |
✨ Finishing Touches
🧪 Generate unit tests (beta)
- Create PR with unit tests
- Commit unit tests in branch
fix/sse-idle-timeout
✨ Simplify code
- Create PR with simplified code
- Commit simplified code in branch
fix/sse-idle-timeout
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/test_sse_idle.py`:
- Around line 19-39: Extend test_events_recovers_from_zombie_stream coverage to
verify the async get_job_events path also recovers from a stalled SSE stream and
reaches the terminal StatusChange without hanging. Add a case using timeout=None
to confirm the explicit no-timeout contract, while preserving the existing
default-timeout recovery assertion and terminal-status checks.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: a37df146-b945-4f32-a970-4e49905d5d6c
📒 Files selected for processing (3)
src/comfy_low/transport.pytests/conftest.pytests/test_sse_idle.py
Extend the zombie-stream regression to the async get_job_events path (same idle timeout + poll fallback) and add a test proving timeout=None disables the idle timeout (rides out the stall until close instead of raising). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Found running a real >10-minute job against prod: the SSE stream went silent at ~3 min (no frames, no keepalive, no close — a "zombie" connection), and
job.events()hung — still blocked minutes after the job had already completed.poll/wait/runwere unaffected (the job itself finished fine).Root cause
ComfyLow.get_job_events(sync + async) opened the SSE stream withtimeout=None, which disables httpx's read timeout — soiter_lines()blocks forever on a stalled-but-open connection. The high-levelJob.events()reconnect/poll-fallback only runs when the stream ends (the generator returns or raises), which never happens for a zombie →events()hangs.Fix
Give
get_job_eventsa read-idle timeout (_SSE_IDLE_TIMEOUT, well above the server's keepalive interval) by default. A stalled stream now raiseshttpx.ReadTimeout, whichJob.events()already catches and turns into a poll (terminal?) / reconnect.timeout=Nonestill available to opt out.Tests
Added a threaded stub
stallmode (sends a couple frames, then holds the socket open and silent) and a regression test assertingevents()recovers via the poll fallback in ~0.3s instead of waiting for close / hanging. Full suite: 74 passed; ruff/format/mypy/hygiene clean.🤖 Generated with Claude Code