refactor: promote atomic_write_text to a shared util; migrate vanilla tmp+replace sites (BE-4353)#595
refactor: promote atomic_write_text to a shared util; migrate vanilla tmp+replace sites (BE-4353)#595mattmillerai wants to merge 5 commits into
Conversation
… tmp+replace sites (BE-4353) Consolidate the duplicated "write via tmp file + os.replace" pattern into a single shared helper `comfy_cli.file_utils.atomic_write_text(path, content, *, fsync=False)` and migrate the vanilla copies onto it: - skills.write_manifest — also FIXES an existing tmp-file leak (an exception between the write and the rename previously left the tmp behind). - skills._atomic_write_text — removed; all four skill writers now use the shared helper. - cql/loader.write_object_info_cache — keeps its best-effort try/except OSError wrapper around the call. - jobs_state.write — uses fsync=True, keeping its locking.file_lock wrapper. - command/workflow._atomic_write_text — removed; a sixth identical copy the ticket's finder overlooked, migrated for completeness. auth/store._write_all is deliberately left untouched: it is a security-hardened superset (O_EXCL + 0o600 at open, 0o700 parent dir, fsync) protecting secrets.
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 8 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughChangesThe PR adds shared atomic text-file persistence with optional durability syncing and failure cleanup. Workflow, cache, job-state, manifest, skill, and rule writers now use it, while tests cover writing, overwriting, cleanup, uniqueness, fsync, and symlink protection. Atomic File Persistence
🚥 Pre-merge checks | ✅ 2✅ Passed checks (2 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
✨ Simplify code
Comment |
There was a problem hiding this comment.
🔍 Cursor Review — Consolidated panel
Triggered by @mattmillerai.
Found 5 finding(s).
| Severity | Count |
|---|---|
| 🟠 High | 1 |
| 🟡 Medium | 2 |
| 🟢 Low | 1 |
| ⚪ Nit | 1 |
Panel: 6/8 reviewers contributed findings.
Reviewers that did not contribute: kimi-k2.5:adversarial (empty), kimi-k2.5:edge-case (empty)
Rewrite the tmp-file creation to use tempfile.mkstemp in the destination directory, addressing the consolidated cursor-review panel findings: - High: drop the PID-only tmp name that let concurrent writers collide on <dest>.<pid>.tmp; mkstemp gives a unique name per write. - Medium (CWE-377): mkstemp opens with O_CREAT|O_EXCL and doesn't follow symlinks, so a pre-planted symlink can't redirect the write. - Medium: fsync now uses the O_RDWR fd from mkstemp, so os.fsync no longer silently no-ops on Windows (FlushFileBuffers needs write access). - Low: except BaseException so a KeyboardInterrupt mid-write still cleans up the tmp file, per the docstring. - Nit: fsync the parent directory after os.replace so the rename itself is durable against power loss. Add tests for unique-per-write tmp names and no-symlink-following. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@comfy_cli/file_utils.py`:
- Around line 39-52: Update the atomic-write flow around tempfile.mkstemp and
os.replace to set the temporary file’s mode before renaming: reuse the existing
destination’s permission bits when the destination exists, otherwise apply the
platform’s umask-derived default. Ensure this occurs before os.replace so the
final file preserves expected shared read access.
In `@tests/comfy_cli/test_file_utils.py`:
- Around line 107-112: Strengthen test_atomic_write_text_fsync_true_still_writes
by spying on the relevant os.fsync/os.fdopen operations, following the existing
mkstemp spy pattern in test_atomic_write_text_tmp_name_is_unique_per_write.
Assert fsync is invoked for both the temporary file descriptor and the parent
directory while preserving the existing content and temporary-file cleanup
assertions.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: b6dcbc15-2421-453c-9cbb-8465a49c9258
📒 Files selected for processing (6)
comfy_cli/command/workflow.pycomfy_cli/cql/loader.pycomfy_cli/file_utils.pycomfy_cli/jobs_state.pycomfy_cli/skills/__init__.pytests/comfy_cli/test_file_utils.py
…ert fsync (BE-4353) CodeRabbit review follow-ups: - mkstemp hardcodes the tmp file to 0600 and os.replace carries that mode onto the destination, so a first atomic write silently stripped group/other read from shared outputs. Restore the existing destination's mode, else the umask-derived default for a new file, before the rename. This matches the pre-migration write_text() behavior of the migrated call sites. - Strengthen the fsync test to spy on os.fsync and assert it's actually invoked on the tmp fd (and the parent dir fd off Windows), plus add mode-preservation tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolves the CodeQL py/overly-permissive-file (high) alert flagged on the atomic-write mode-preservation test. The test only needs a non-0600 mode to prove mkstemp's hardcoded 0600 does not clobber the destination mode; 0o640 keeps the group-read regression guard without granting world read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…t (BE-4353) The prior 0o640 (world→group) still trips py/overly-permissive-file because group-read is flagged too. The test only needs a non-0600 mode to prove mkstemp's hardcoded 0600 doesn't clobber the destination mode, so use owner-execute (0o700) — a bit 0600 lacks, with no group/other bits — keeping the required CodeQL check green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Fixed in e8ae317: the mode-preservation test now sets |
ELI-5
Several places in the CLI write a file "safely" the same way: write to a temporary file, then rename it over the real file so a Ctrl-C mid-write can never leave a half-written or empty file. That same ~10-line snippet was copy-pasted into five (really six) different modules, and one of the copies had a small bug — it leaked the temp file if something failed between writing and renaming. This PR pulls that pattern into one shared helper,
comfy_cli.file_utils.atomic_write_text(...), and points the vanilla copies at it (fixing the leak in the process). The one specially-hardened copy that protects secrets is left alone.What changed
New shared helper in
comfy_cli/file_utils.py:tmp-in-same-dir +
os.replace, cleans up the tmp file and re-raises on any failure, creates parent dirs, and optionallyfsyncs the tmp contents before the rename (best-effort, matching prior per-site behavior).Migrated call sites:
skills.write_manifestwrite_textandos.replaceleaked the tmp file.skills._atomic_write_text_write_claude_skill,_write_cursor_rule,_upsert_agents_md_block×2) now call the shared helper.cql/loader.write_object_info_cachetry/except OSErrorwrapper around the call; its own tmp-cleanup is now handled by the helper.jobs_state.writefsync=True, keeping itslocking.file_lockwrapper. Drops the now-redundantsecrets.token_hextmp suffix (writes are already serialized per-path by the lock).command/workflow._atomic_write_textDeliberately untouched:
auth/store._write_all— a security-hardened superset (O_CREAT|O_EXCLwith0o600at open time,0o700parent dir, fsync) protecting secret material. Folding it into a generic helper would dilute those permission guarantees for zero benefit; the ticket explicitly excludes it.tmp-name normalization (the riskiest change)
write_manifestandjobs_statepreviously replaced the suffix (manifest.json→manifest.<pid>.tmp); the shared helper appends (manifest.json.<pid>.tmp). This is harmless — tmp names are ephemeral and renamed away immediately — and importantly both old and new names end in.tmp, so neither matches the*.jsonglob thatcommand/jobs.pyuses to list job state files. Verified: no job-listing glob or watcher can observe the tmp under either scheme.Tests
atomic_write_textintests/comfy_cli/test_file_utils.py: creates file + parents, overwrites existing,fsync=Truepath, and tmp-cleanup + destination-untouched on a failed rename.ruff check/ruff format --checkclean on all changed files. (The 15 pre-existingUP038ruff findings in untouched files are onmainalready, unrelated to this change.)Judgment call
The ticket enumerated five duplicate sites and migrated three (+ optional
jobs_state). While auditing I found a sixth identical_atomic_write_textincommand/workflow.pythat the finder overlooked (not a deliberate exclusion likeauth/store). Since the ticket's whole purpose is to consolidate these vanilla duplicates, I migrated it too. It is semantically identical; the only difference is the shared helper adds aparent.mkdir(parents=True, exist_ok=True), which is a harmless no-op at both of its call sites (both already write into pre-existing directories). Flagging it here since it's beyond the ticket's literal list.