Agent-state monitor over the experimental control-mode engine#692
Open
tony wants to merge 56 commits into
Open
Agent-state monitor over the experimental control-mode engine#692tony wants to merge 56 commits into
tony wants to merge 56 commits into
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## engine-ops #692 +/- ##
==============================================
+ Coverage 75.23% 77.31% +2.08%
==============================================
Files 225 267 +42
Lines 13638 16002 +2364
Branches 1784 2037 +253
==============================================
+ Hits 10260 12372 +2112
- Misses 2690 2859 +169
- Partials 688 771 +83 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
tony
added a commit
that referenced
this pull request
Jun 28, 2026
why: A PR does not own its release version; the monitor entry named a concrete version that also went stale after rebasing onto newer master. what: - Open the entry with the package as subject, not "libtmux X.Y.Z ships" - Add the (#692) PR ref to the deliverable heading
tony
added a commit
that referenced
this pull request
Jul 4, 2026
why: A PR does not own its release version; the monitor entry named a concrete version that also went stale after rebasing onto newer master. what: - Open the entry with the package as subject, not "libtmux X.Y.Z ships" - Add the (#692) PR ref to the deliverable heading
tony
added a commit
that referenced
this pull request
Jul 4, 2026
why: A PR does not own its release version; the monitor entry named a concrete version that also went stale after rebasing onto newer master. what: - Open the entry with the package as subject, not "libtmux X.Y.Z ships" - Add the (#692) PR ref to the deliverable heading
tony
added a commit
that referenced
this pull request
Jul 4, 2026
why: A PR does not own its release version; the monitor entry named a concrete version that also went stale after rebasing onto newer master. what: - Open the entry with the package as subject, not "libtmux X.Y.Z ships" - Add the (#692) PR ref to the deliverable heading
tony
added a commit
that referenced
this pull request
Jul 5, 2026
why: A PR does not own its release version; the monitor entry named a concrete version that also went stale after rebasing onto newer master. what: - Open the entry with the package as subject, not "libtmux X.Y.Z ships" - Add the (#692) PR ref to the deliverable heading
tony
added a commit
that referenced
this pull request
Jul 5, 2026
why: A PR does not own its release version; the monitor entry named a concrete version that also went stale after rebasing onto newer master. what: - Open the entry with the package as subject, not "libtmux X.Y.Z ships" - Add the (#692) PR ref to the deliverable heading
tony
added a commit
that referenced
this pull request
Jul 5, 2026
why: A PR does not own its release version; the monitor entry named a concrete version that also went stale after rebasing onto newer master. what: - Open the entry with the package as subject, not "libtmux X.Y.Z ships" - Add the (#692) PR ref to the deliverable heading
tony
added a commit
that referenced
this pull request
Jul 5, 2026
why: A PR does not own its release version; the monitor entry named a concrete version that also went stale after rebasing onto newer master. what: - Open the entry with the package as subject, not "libtmux X.Y.Z ships" - Add the (#692) PR ref to the deliverable heading
tony
added a commit
that referenced
this pull request
Jul 5, 2026
why: A PR does not own its release version; the monitor entry named a concrete version that also went stale after rebasing onto newer master. what: - Open the entry with the package as subject, not "libtmux X.Y.Z ships" - Add the (#692) PR ref to the deliverable heading
why: The shared vocabulary every agents module reads/writes. what: - AgentState (running/awaiting_input/idle/exited/unknown) with from_signal - frozen Agent record + is_running/is_awaiting helpers
why: Out-of-order/replayed agent-state updates must converge to newest. what: - Stamp(counter, writer) with deterministic tie-break - latest() guard + pluggable Clock + MonotonicCounter default
why: When the display-message own-id probe fails, own is None, so the `sid != own` guard is always true and _primary_session_id falls through to return ids[0] — tmux's phantom `tmux -C` session (it sorts first), which holds no agent panes. Attaching there leaves the option channel effectively silent. what: - Return None from _primary_session_id when the own-session probe fails, so start() skips attach instead of binding to the phantom session - Cover the case with a fake engine whose display-message probe raises
why: tmux reports a failed attach (e.g. stale session id) as a non-zero returncode, not an exception, so the monitor recorded _attached_session even when the attach failed -- silencing the option channel. what: - monitor.start() now records the sticky attach only when attach-session returns returncode 0; logs the stderr on failure - document _replay_attach's optimistic fire-and-forget attach and that a failed re-attach self-corrects on the monitor's next reconcile
why: The agent monitor needs to tell a floating overlay (e.g. a status
HUD) apart from a real agent pane; the snapshot had no floating flag and
the monitor's pane format did not request it.
what:
- Add PaneSnapshot.floating, parsed from #{pane_floating_flag} (tmux
3.7+; renders empty -> False on older tmux, so no version gate)
- Request pane_floating_flag in the monitor's PANE_FORMAT
- Cover the snapshot floating flag and the format request
Foundation for the floating HUD; the renderer + monitor self-exclusion
land next.
why: The agent monitor was observation-only; on tmux 3.7 a floating overlay can surface live agent state in the session itself, with no external UI. what: - Add HudRenderer: a pure AgentStore -> text frame plus the typed RespawnPane paint op (the frame is shell-quoted and held open with `tail -f /dev/null` so it persists between repaints) - AgentMonitor gains opt-in `hud=True`: start() creates one floating NewPane over the primary session and captures its id; the supervised drain repaints on every store change (dirty flag set in _observe and reconcile); stop() kills the HUD pane - Exclude the HUD's own pane from _reconcile_once so it never enters the diff or the health sweep (tracking is signal-driven, so this is the only exclusion needed); best-effort throughout (no session, engine error, or tmux < 3.7 silently skips the HUD) - Cover the renderer, the repaint op, HUD create/teardown, and the reconcile self-exclusion
why: watch_agents opened its own engine.subscribe() and re-ingested the fan-out stream while the monitor's drain already ingests it, so every event was processed twice -- drifting the MonotonicCounter and `since` stamps. It was also tagged readOnlyHint=True despite mutating the store. what: - Drop the redundant subscribe()+ingest loop; observe the monitor's live store over the window (the monitor's drain is the sole ingester), so readOnlyHint=True is now accurate - Cover that watch_agents never calls engine.subscribe()
why: _ensure_hud ran only in start(). After a full tmux restart the HUD pane id is dead; _repaint_hud ignored the arun result (which reports a tmux-side failure as data, not a raise), so _hud_pane_id was never cleared and the HUD stayed dark for the rest of the session. what: - _repaint_hud now checks the result: a failed/errored repaint drops _hud_pane_id (leaving it dirty) so it can be recreated; the dirty flag is cleared only on a successful paint - _run recreates the HUD (_ensure_hud) after a reconcile when it is enabled but has no pane id -- covering both a restart and an initial create that had no session yet - Cover ok-keeps-pane vs fail-drops-pane (parametrized)
why: The module docstring said the HUD "repaints on every store change", but _hud_dirty is set unconditionally on each notification and reconcile, so it repaints on those events rather than only on an actual mutation. what: - Reword to "repaints after each notification and reconcile"
why: The AgentHook.install/uninstall doctests ran a bare ClaudeCodeHook(), which defaults settings_path to the real ~/.claude/settings.json and rewrites it on every doctest run. The "no-op on stub" comment was wrong -- these methods always write. what: - Redirect each doctest into a tempfile.TemporaryDirectory(), mirroring the already-isolated claude.py/codex.py doctests
why: The agent-monitor entry described the liveness/reconcile sweep as "a periodic health probe" running "every few seconds". It is actually event/reconnect-driven: _run reconciles at startup and again each time the subscribe stream ends (a disconnect/reconnect). The only sleep is a retry backoff on the failure path, not a timer. what: - "refreshed by a periodic health probe" -> "refreshed on each reconciliation" - "runs every few seconds" -> "runs at startup and on each engine reconnect"
why: Two monitor test fakes failed strict mypy: a stream-engine fake missing the _StreamEngine Protocol's _attached_session member, and a capturing FastMCP stand-in passed where FastMCP is expected. what: - Add _attached_session to FakeStreamEngine/_BlockingStreamEngine - Cast the capturing MCP fake to FastMCP at the register_agents call
why: from_dict raised ValueError on a state string absent from the current enum (e.g. a store written by a newer version), crashing the monitor on startup when it loads the store. what: - Deserialize state via AgentState.from_signal (unknown -> UNKNOWN), mirroring signal ingestion - Add parametrized tests for known/unknown/garbage states
why: `emit ... --name` with the flag as the final CLI arg raised IndexError instead of a clean exit, surfacing a traceback to the agent's hook runner. what: - Fall back to name=None when --name has no following value - Add parametrized tests for main's --name parsing
why: install_agent_hooks awaited blocking file I/O (read, fsync, atomic replace) directly on the event loop, stalling concurrent MCP tools during an install. what: - Run hook install/status via asyncio.to_thread - Add parametrized tests for the install tool (known/unknown agent)
why: A PR does not own its release version; the monitor entry named a concrete version that also went stale after rebasing onto newer master. what: - Open the entry with the package as subject, not "libtmux X.Y.Z ships" - Add the (#692) PR ref to the deliverable heading
why: The monitor knew agent state but gave no way to act on it — no blocking wait for a state, no safe driver, and concurrent sends to one pane interleaved keystrokes. These are the fan-in/fan-out verbs an orchestrator needs, plus the correctness primitives single-operator tools never provide. what: - agents/wait.py: wait_for_agent_state + fleet wait_for_agents over a pure WaiterRegistry woken from the monitor's single _observe edge; outcomes as data (AgentWait/WaitReason); zero tmux calls (level check + future, never a poll). - agents/drive.py: send_to_agent/send_to_agents fold every multi-step and fleet send into ONE dispatch via LazyPlan + FoldingPlanner; a process-wide per-pane pane_lock chokepoint serializes all async keystroke injection; DedupLedger makes a keyed retry a no-op. - agents/state.py: AgentTransition value. - agents/monitor.py: host the registry/ledger; diff before->after in _observe/_apply_health/reconcile to wake waiters, emit a structured agent_* INFO log, and fan AgentTransition to observers; stop() resolves parked waits (STOPPED) so none hang. - mcp: wait_for_agent + send_to_agent tools; asend_input now takes the same pane_lock (the chokepoint retrofit). - Design spec + unit/live tests (folding asserted at one dispatch).
why: "facade" is generic jargon. The replacement is grounded in real Python package-naming convention rather than invented: "wrappers" is the word Flask (wrappers.py: Request/Response over base classes), PyTorch (torch/_prims_common/wrappers.py), and the stdlib (io.TextIOWrapper) use for "an object that wraps a lower thing and adds ergonomics" — which is exactly what these engine-bound, mode-in-the-type classes are, and what their own docstrings already called them. Rejected alternatives: "handles" collides with asyncio.Handle (the very runtime an AsyncPane lives in), file handles, and logging Handlers; "orm" collides with describing the whole library as a typed ORM; "proxies" has only class-level pedigree (no studied lib names a *package* proxies) and mis-signals (these build and run typed Operations, they do not transparently forward to a referent) — and types.MappingProxyType already means "read-only view" one directory over in ops/. what: - git mv experimental/facade -> experimental/wrappers (+ the test dir and the two scope-named test files). - Rewrite imports experimental.facade -> experimental.wrappers and reword the layer's prose (facade/handle -> wrapper) in docstrings and docs/experimental. - Add the "typed proxy over the Core spine" lineage (the SQLAlchemy AsyncSession parallel) to the package docstring, where the insight belongs rather than in the directory name. - Class names (Eager*/Lazy*/Async* x Server/Session/Window/Pane/Client) are unchanged. Naming chosen by a 10-agent study of SQLAlchemy/Django/CPython/tmux and the broader ecosystem under ~/study, with adversarial review.
why: an orchestrator wants the fleet's state visible at a glance, and the references' most common surface is the tmux status line. We can render it as an "instantly rendering UI" -- read every agent's state from the in-process store (zero tmux calls) and paint the whole fleet with a SINGLE set-option. what: - Add experimental/agents/statusline.py: pure render_status_line(agents) -> compact per-state tally in attention order (labels overridable); status_line_op() building the lone set-option; and async paint_status_line(engine, source) that reads agents from a monitor (0 calls) or a pure sequence and dispatches exactly one set-option for status-right. - A tmux-native render surface parallel to the floating HudRenderer; owns the mechanism (read at zero cost -> one dispatch), not the format. - Export from the agents package. Tests: pure tally + empty + custom labels + op shape + a recorder proving exactly one write + a live read-back of status-right against real tmux.
why: the sync ControlModeEngine has the same throwaway-session leak as the async one -- a bare `tmux -C` connect implies new-session, littering the target server. Fix both engines so the defect is gone everywhere, not just the async path the MCP happens to use. what: - Add ControlModeEngine._reap_own_session(): after connect (still on the phantom, before any attach), set `destroy-unattached on` on the current session via the low-level _write/_read_blocks inside _ensure_started's locked section, reading and discarding its result block so it can't desync the next command (a re-entrant self.run() would deadlock on _lock). - Live tests mirroring the async ones: the phantom carries destroy-unattached and is reaped on close.
why: the settle monitor is deliberately needle-free, but "wait until the pane prints READY / the sentinel line appears" is a distinct, common await -- a server coming up, a build banner, a prompt. Resolving the instant it appears returns sooner than waiting for the idle gap (the "instantly" north star). what: - accumulate_until_settle gains an optional `needle` (a str matched literally, or a compiled re.Pattern searched) and a new `matched` SettleReason; it short- circuits the moment the accumulated output matches, before the idle/byte/time caps. None keeps the pure needle-free settle unchanged. - Thread `needle` through the MCP `wait_for_output` tool. - Tests: literal, regex, cross-chunk match, and no-needle-still-settles; doctest.
why: The experimental user-facing domain layer should use Pythonic object naming and avoid facade or handle terminology. The branch has not shipped this import surface, so the clean package name can land without a compatibility alias. what: - Rename libtmux.experimental.wrappers to libtmux.experimental.objects - Update imports, tests, and package prose for the object surface - Add an import guard for the experimental objects package roots
why: Agents need a one-call path for running shell commands in panes and observing the live result. Composing send and wait behind the pane drive lock avoids extra backend calls and prevents interleaved input. what: - Add run_in_pane over the existing SendKeys operation and output monitor - Extract the wait_for_output monitor body for reuse - Register and document the new MCP tool in prompts, instructions, and docs
why: Agent orchestration needs a first-class state for completed turns that need review, distinct from idle shells and approval waits. what: - Add AgentState.DONE to parsing, rollups, status-line rendering, and waits - Emit done from Claude/Codex turn-complete hooks and keep done send-ready - Cover store, MCP, hook, wait, drive, and query handling for done
why: Agents can now submit and build a family of declarative workspaces in one backend call while keeping the chainable engine and planner in charge of dispatch cost. what: - Add WorkspaceSet compile/build APIs with SlotRef and host-step rebasing - Add workspace_status projections over snapshots and agent state - Expose build_workspaces through MCP and document the new workspace-set flow
why: Raw tmux %output notifications are silent until the control-mode client attaches to a session, so output-source event tools could appear healthy while returning empty streams. what: - Add target-aware attachment for watch_events and poll_events in output mode - Report a visible tmux://events error before an output-source pull stream is attached - Cover push, pull, resource, and live raw-output event paths
why: Agent monitoring already has live state, hooks, and synchronization primitives, but users need a tmux-like command that boots or reattaches the agent interface without manual setup. what: - Add top-level libtmux.agents module and libtmux-agents console script - Build the managed console session through the declarative workspace engine - Add status, hooks, wait, and send CLI verbs backed by existing agent state - Document the console workflow and cover detached boot/status behavior
why: The console dashboard could stay on "(no agents)" because the CLI opened the control-mode connection before registering the monitor subscription, and reconcile did not seed pane options that were already present. what: - Register CLI monitor subscriptions before the first control-mode connection - Include durable @agent_state and @agent_name values in pane reconciliation - Add live and reducer coverage for startup and after-start state changes - Harden the live MCP output test against tmux chunk boundaries
why: The console can start before Claude or Codex lifecycle hooks emit @agent_state, leaving visible agent panes absent from the dashboard. what: - Seed weak process-discovered agent records from pane commands and local descendants - Let explicit option or OSC state override process-discovered records - Mark process-discovered agents exited when the agent command disappears - Add reducer, process-table, and missing-process-table coverage
why: rebasing the agent-monitor branch onto the fluent engine-ops merged two divergent versions of query.py and the control engines. The mechanical rebase resolution kept engine-ops's split-type pane handles and supatui's fuller async engine; this restores what each side dropped. what: - Re-graft the agents() query (AgentQuery/agents()/ATTENTION/_query_agents) onto the split-type query.py - Re-add tmux_version() to both control engines (kept supatui's engine, which lacked it) - Export workspace_status from the workspace package
why: The base engine-ops branch renamed the in-memory engine to MockEngine (from the former name); the agent monitor, statusline, and drive examples plus their tests still constructed the old async class. what: - Update the async in-memory engine references to AsyncMockEngine in agents doctests (monitor, statusline, drive) and their tests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
libtmux.experimental.agents— a headless, self-healing monitor that reports, per pane, which coding agent isRUNNING,AWAITING_INPUT,IDLE,EXITED, orUNKNOWN, without polling or scraping pane output. Answers the orchestration question "which of my parallel agents needs me right now?"@agent_state→%subscription-changed) and a remoteOSC 3008escape carried in control-mode%output— reconciled by a lock-free last-writer-wins merge so the two channels can race freely.AsyncControlModeEnginewith a supervised reconnect loop, a death-sentinel that closes subscribers cleanly, and per-subscriber broadcast queues, so atmuxrestart or socket blip self-heals and concurrent consumers never steal each other's events.refresh-client -B/-Cto theRefreshClientoperation — the substrate the monitor subscribes through (debounced, server-side change detection) instead of raw%output.list_agents,watch_agents, andinstall_agent_hookstools, plus a lifespan that starts/stops the monitor and gates the tools behind an engine capability check.~/.claude/settings.json) and Codex (~/.codex/config.toml), installable into a running session at any time.Stacks on #690. Everything is additive under
libtmux.experimental— no existing public API is touched, and the package is mypy-strict clean.Changes by area
src/libtmux/experimental/agents/(new package)state.py: theAgentStateenum (withfrom_signal()that maps unknown values toUNKNOWNrather than raising) and the frozenAgentrecord (name,state,since,source,pid,alive, plusis_running/is_awaiting).merge.py:Stamp(counter, writer)ordering andlatest()— the convergent, idempotent, out-of-order-tolerant merge rule. The clock is a pluggable callable (MonotonicCounternow, HLC later).store.py: the durable value tier — a frozenAgentStore, the pureapply(store, event, *, now)reducer,Observed/Vanishedevents, aStorageprotocol, andJsonFile(atomic temp-write + rename). Onlyapply()mutates state.signals.py:OptionSignal(local) andOscSignal(remote) — the latter reassembles the byte-fragmentedOSC 3008stream tmux delivers in%output.health.py:is_alive(pid)viaos.kill(pid, 0);None(a PID-less remote pane) is treated as alive so a remote agent is never falsely expired.tree.py:panes_of()/diff_panes()derived frommodelssnapshots — the live session→window→pane projection.monitor.py:AgentMonitor— the supervisedstart/stop/reconcile/status/agentscontract, the reducer pipeline, and the reconcile sweep that self-attaches (excluding its owntmux -Csession) and self-heals across reconnects.hooks/: theAgentHookprotocol (detect/install/uninstall/status), the sharedemit()(localset-option, else remoteOSC 3008→/dev/tty),ClaudeCodeHook,CodexHook, and aregistry().src/libtmux/experimental/engines/async_control_mode.py_supervisor, deterministic jittered_backoff, backoff reset on a healthy connect), desired-state replay (add_subscription/set_attach_targets→_replay_subscriptions/_replay_attach), a_STREAM_ENDdeath-sentinel broadcast to per-subscriber queues, and asubscribe()that no longer hangs when called afteraclose().src/libtmux/experimental/ops/_ops/refresh_client.pysubscribe(-B) andsize(-C) parameters, version-gated and serializable like every other operation.src/libtmux/experimental/mcp/vocabulary/agents.py:register_agents()registerslist_agents/watch_agents/install_agent_hooks, behind asupports_monitor()capability gate._lifespan.py,events.py,fastmcp_adapter.py: start/stop the monitor in the server lifespan and skip the agent tools when the lifespan won't bring a monitor up.CHANGES### What's new.Design decisions
list-*snapshot diff is the correctness backstop, because tmux's change feed has blind spots (pane-died, window-resized, and title changes emit no notification).show-options -p -vself-heals a dropped notification) but slow (~1 s debounce); the OSC path is instant and the only one that survives SSH (a remoteset-optioncan't reach the local socket) but rides the lossy%outputstream. Neither alone is sufficient.OptionSignal@agent_state→%subscription-changedOscSignalOSC 3008→%output(byte-fragmented)EXITED. Local panes expire only on a failed PID probe; PID-less remote panes stay at their last-known state.(counter, writer)latest()guard runs before the coalescing value write, so the two channels merge deterministically without locks regardless of arrival order.AsyncTmuxEngineprotocol, themodelssnapshots, and aStoragesink — not onServer/Session/Window/Pane.list_agentswatch_agentsinstall_agent_hooksVerification
The package is decoupled from the classic ORM (depends only on the engine protocol + models):
$ rg -n "from libtmux\.(server|session|window|pane|common) import|import libtmux\b" src/libtmux/experimental/agentsThe remote signal is written to the pane tty, not stdout (which agent hooks capture):
$ rg -n "/dev/tty" src/libtmux/experimental/agents/hooks/emit.pyAsync tests follow the repo convention (
asyncio.run, nopytest-asyncio):$ rg -n "pytest_asyncio|pytest\.mark\.asyncio" tests/experimental/agents tests/experimental/engines/test_async_control_mode_supervisor.pyMCP agent tools are gated behind an engine capability check:
$ rg -n "supports_monitor" src/libtmux/experimental/mcpTest plan
state/merge/store/signals/tree/healthunit tests + doctests (reducer idempotence, byte-fragmented OSC reassembly, last-writer-wins).test_async_control_mode_supervisor.pyandtest_async_control_mode_sentinel.pycover supervised reconnect, attach replay on reconnect, broadcast to concurrent subscribers, and subscribe-after-close.test_live_monitor.pyvalidates self-attach, reconcile, and survives-reconnect against a real tmux server via the libtmux fixtures.notifyfallback.test_agents_tools.py/test_attach_reset.pycover tool registration and the capability gate.ruff check,ruff format --check,mypy src tests,pytest, andjust build-docs(the CHANGES entry + catalog directive) all green.Refs #688, #689. Builds on #690.