test(run-engine): address CK review (real age scenarios, honesty)

Blind two-model review found the ckSkew/ckBalanced baseline starvation was a
Redis lexicographic tie-break on equal enqueue timestamps, not age order.
Redesigned scenarios so the backlog key keeps a genuinely old head and other
keys arrive via poisson, so the baseline now exercises real age-order
starvation (ckSkew baseline 0.187, worst key waits 1321ms; SFQ cuts it to 16ms).
FINDINGS corrected: lead with per-key wait (contention share is volume-confounded
for low-volume keys), and add the key fidelity limit the review surfaced: the
harness serves one key per Lua call, while production batches and re-scores a
served key mid-call, so this proves the ordering fix only at maxCount=1 and a
production fix must advance per-key state inside the batched Lua. Softened
'viable' to 'worth a design spike'.

Author: 1stvamp

internal-packages/run-engine/src/run-queue/fairness-spike-ck/FINDINGS.md

@@ -2,98 +2,121 @@
 
 Throwaway spike. Ships nothing; delete before any merge to main.
 
-Bottom line: the base-queue spike's ranking carries over to the real seam, and
-more starkly. At the concurrency-key grain the production baseline (serve the
-oldest-head CK first) starves other keys outright, and virtual-time (SFQ) and
-stride fix it. DRR is close. A CoDel wrapper on top of the broken baseline makes
-it worse. Crucially, this was measured by driving the real
-`dequeueMessagesFromCkQueueTracked` Lua and its real concurrency gating; only the
-`ckIndex` scores were rewritten to express each discipline, so the result says a
-production fix is viable (advance the ckIndex score by a fair discipline instead
-of by head timestamp).
-
-## What was driven, and the one affordance
-
-Runs enqueue across many concurrency keys under a single base queue via the real
-`RunQueue`. The per-CK pick is `ZRANGEBYSCORE ckIndexKey -inf now` inside the CK
-Lua (lowest score first, score = head timestamp). For candidates the driver
-rewrites those scores each round to encode the discipline's order; the baseline
-leaves them as the Lua maintains them, which is production behaviour. Enqueue,
-dequeue, per-CK and env concurrency, and ack all run through the real code. The
-rescore sweep and an `onServiced(key)` hook are the only affordances; in
-production that advance would live in the Lua that maintains `ckIndex`. A smoke
-test confirms the baseline path reproduces age-order starvation.
-
-Fairness is measured as in the base-queue spike (arrival-aware contention share
-and per-key wait), carrying forward its corrected metric and monotonic-floor
-disciplines. All keys are equal weight (concurrency keys carry no configured
-weight in production).
+Bottom line: the base-queue spike's direction carries over to the real seam. At
+the concurrency-key grain the production baseline (serve the oldest-head CK
+first) starves keys that arrive behind a big backlog, and virtual-time (SFQ) and
+stride fix it: they cut the starved key's wait from ~1300ms to ~20ms by making
+the backlog key wait its turn. DRR does the same. A CoDel wrapper on the baseline
+makes it worse. This was measured by driving the real
+`dequeueMessagesFromCkQueueTracked` Lua and only rewriting `ckIndex` scores to
+express each discipline. That is enough to say the ordering fix is worth a design
+spike, but NOT that a production implementation is proven (see the fidelity
+caveat: the spike serves one key per Lua call, and production dequeues in
+batches).
+
+## What was driven, and the two things to know before reading numbers
+
+Runs enqueue across many concurrency keys under one base queue via the real
+`RunQueue`. The per-CK pick is `ZRANGEBYSCORE ckIndexKey -inf now` in the CK Lua
+(lowest score first, score = head timestamp). Candidates rewrite those scores
+each round to encode discipline order; the baseline leaves them (production age
+order). Enqueue, dequeue, concurrency gating and ack all run through the real
+code.
+
+Two caveats a review forced, both load-bearing:
+
+1. Lead with wait, not contention share. The contention-share metric is
+   volume-confounded for low-volume keys (a key with 15 runs cannot take a third
+   of a long window even when served instantly), so on these scenarios it lands
+   around 0.7 to 0.9 for a discipline that has in fact eliminated the starvation.
+   The per-key wait is the clean signal.
+2. The scenarios must give keys genuinely different head ages. An earlier version
+   enqueued every run at one timestamp; with tied `ckIndex` scores the real Lua
+   falls back to a lexicographic member-name tie-break, so the "baseline starves
+   the heavy key's rivals" result was actually "Redis sorts by name" and the
+   heavy key only won because "heavy" sorts before "light". Fixed: the backlog
+   key fires at once (persistently old head) and the other keys arrive via
+   poisson (distinct, later heads), so the baseline now exercises real age order.
 
 ## Results
 
-`contWorstS/W` mean over 3 seeds (min..max). Higher is fairer.
+`contWorstS/W` mean over 3 seeds (min..max), and the worst-served key's mean wait
+(seed-a, logical ms). Read the wait column as the headline.
 
-| scenario    | baseline            | sfq                 | drr                 | stride | codel(sfq) | codel(baseline) |
-| ----------- | ------------------- | ------------------- | ------------------- | ------ | ---------- | --------------- |
-| ckSkew      | 0.000               | 0.982 (0.946..1.000)| 0.974               | 0.982  | 0.982      | 0.000           |
-| ckBalanced  | 0.000               | 0.987               | 0.987               | 0.987  | 0.987      | 0.000           |
-| ckTrickle   | 0.279 (0.254..0.291)| 0.909 (0.891..0.918)| 0.790 (0.769..0.818)| 0.909  | 0.909      | 0.018           |
+| scenario   | discipline | contWorstS/W        | worst-key wait | backlog-key wait |
+| ---------- | ---------- | ------------------- | -------------- | ---------------- |
+| ckSkew     | baseline   | 0.187 (0.186..0.188)| 1321           | 872              |
+| ckSkew     | sfq        | 0.723 (0.655..0.769)| 16             | 1150             |
+| ckSkew     | drr        | 0.608 (0.556..0.648)| 21             | 1149             |
+| ckTrickle  | baseline   | 0.279 (0.254..0.291)| 1339           | 872              |
+| ckTrickle  | sfq        | 0.909 (0.891..0.918)| 24             | 1237             |
+| ckTrickle  | drr        | 0.790 (0.769..0.818)| 30             | 1236             |
 
-Per-key mean wait (seed-a, logical ms):
+Full matrix (contWorstS/W mean over 3 seeds):
 
-| scenario / discipline | starved-key wait | bulk-key wait |
-| --------------------- | ---------------- | ------------- |
-| ckSkew baseline       | 1853             | 872           |
-| ckSkew sfq            | 265              | 1328          |
-| ckTrickle baseline    | 1252             | 872           |
-| ckTrickle sfq         | 22               | 1237          |
-
-Baseline age-order drives a light key's contention share to 0 on ckSkew (it is
-served only after the heavy key drains) and its wait to 1853ms; SFQ cuts that to
-265ms, and on ckTrickle from 1252 to 22ms, by making the bulk key wait its turn.
+| scenario   | baseline            | sfq                 | drr                 | stride | codel(sfq) | codel(baseline)     |
+| ---------- | ------------------- | ------------------- | ------------------- | ------ | ---------- | ------------------- |
+| ckSkew     | 0.187               | 0.723               | 0.608               | 0.723  | 0.723      | 0.104 (0.000..0.157)|
+| ckBalanced | 0.515 (0.444..0.600)| 0.611 (0.462..0.800)| 0.730 (0.615..0.909)| 0.611  | 0.611      | 0.464               |
+| ckTrickle  | 0.279               | 0.909               | 0.790               | 0.909  | 0.909      | 0.018 (0.000..0.055)|
 
 ## Verdict per discipline (at the concurrency-key grain)
 
-- SFQ / stride: fix the starvation (about 0.98 on skew/balanced, 0.909 on
-  trickle), seed-stable, and cut the starved key's wait 7x (skew) to 57x
-  (trickle). Same as the base-queue spike. Recommended.
-- DRR: close behind (0.974 skew, 0.987 balanced, 0.790 trickle). The trickle gap
-  is the same batch-drain interaction noted in the base-queue spike.
-- CoDel(sfq): no harm here, matches SFQ. Unlike the base-queue trickle scenario
-  it did not overshoot, but it also added nothing; the fair base already bounds
-  sojourn.
-- CoDel(baseline): harmful. Hoisting stale keys on top of the broken baseline
-  drove ckTrickle to 0.018 (worse than baseline's 0.279). A staleness monitor is
-  not a substitute for a fair base.
-- Baseline (production age order): starves keys at this grain. Worse than the
-  base-queue proxy (0.000 vs 0.288 there), because a concurrency key with a
-  backlog of old heads is served to exhaustion before newer keys get a turn. This
-  is #2617 measured at the seam where it actually lives.
-
-## Caveats
-
-- The rescore sweep stands in for a production change to how `ckIndex` scores are
-  maintained. The spike proves the ordering fix works through the real dequeue
-  Lua and concurrency gating; it does not implement or measure the production
-  wiring (which would advance the score inside the enqueue/dequeue Lua and hold
-  per-key discipline state in Redis, not process memory).
-- ckBalanced baseline reading 0.000 is partly a tie-break artifact: with equal
-  enqueue timestamps the age order is a deterministic member-name tie-break that
-  parks one key last during a short contention window. The direction (baseline
-  unfair, candidates fair) is right; lean on ckSkew and the wait numbers for the
-  magnitude.
+- SFQ / stride: fix the starvation. Contention share improves (skew 0.187 to
+  0.723, trickle 0.279 to 0.909) and the starved key's wait collapses (skew 1321
+  to 16, trickle 1339 to 24) because the backlog key now waits its turn (its wait
+  rises 872 to ~1150 to 1237). Identical to each other on every scenario.
+  Recommended discipline for the fix.
+- DRR: fixes the wait just as well (skew 21, trickle 30) and its contention share
+  tracks SFQ within noise (sometimes a little lower, sometimes higher, e.g.
+  balanced 0.730 vs 0.611). Fine.
+- CoDel(sfq): no harm, matches SFQ to the decimal. Adds nothing on top of a fair
+  base.
+- CoDel(baseline): harmful. Hoisting stale keys on top of the age-order baseline
+  drove ckSkew to 0.104 (below baseline's 0.187) and ckTrickle to 0.018 (below
+  0.279). A staleness monitor is not a substitute for a fair base.
+- Baseline (production age order): starves keys that queue behind a backlog
+  (ckSkew 0.187, worst key waits 1321ms; ckTrickle 0.279, 1339ms). It is roughly
+  fair when keys are symmetric (ckBalanced 0.515, though seed-variant 0.444 to
+  0.600). This is the #2617 dynamic at the seam where it lives.
+
+## Fidelity caveat (the reason this is not "proven for production")
+
+The driver dequeues one key per Lua call (`maxCount = 1`) and rescores `ckIndex`
+before each call. Production dequeues in batches (`maxCount` default 10). Inside a
+batched CK-dequeue call the Lua re-scores each served key back to its head
+timestamp as it goes, so a once-per-round rescore would only steer the FIRST pick
+of a batch; the rest would follow head-timestamp order again. So this spike
+demonstrates the ordering fix only in a one-key-per-call regime, which is not how
+production dequeues. A real fix has to advance per-key discipline state inside the
+Lua on every serve (and hold that state in Redis, not process memory). This spike
+does not exercise that batch path, so the correct claim is "the ordering fix is
+worth a design spike", not "a production fix is viable".
+
+## Other caveats
+
+- Contention share is volume-confounded (see above); the wait column is the
+  trustworthy signal, and the contention numbers should be read as directional.
+- The DRR contention-share gap is NOT the base-queue spike's batch-drain artifact
+  (that harness batched; this one serves one key per call and advances DRR's
+  deficit every serve). The cause of DRR's slightly lower share here is not
+  established; its wait result is as good as SFQ's.
+- Per-CK concurrency gating never binds in these runs (no per-CK limit is set, so
+  it collapses to the env limit), so the spike says nothing about the per-CK
+  concurrency-limit-multiplication half of #2617, which is out of scope.
+- A rescore discipline advances its floor/ring state on the final no-op drain
+  round of an instant (order() is called before the empty dequeue). It is
+  self-correcting and does not corrupt the event-based metrics, but it is a minor
+  infidelity to a production per-serve advance.
 - Equal weights only; single shard, single base queue, single sequential
-  consumer; simulated holds on a logical clock; 3 seeds (shows the baseline's
-  failure and the virtual-time schemes' stability, not a statistical study).
-- The other half of #2617, per-CK concurrency-limit multiplication, is a limit
-  problem not a dequeue-ordering one, and is out of scope here.
+  consumer; simulated holds on a logical clock; 3 seeds.
 
 ## Recommended direction
 
-The fix for #2617 at the CK grain is to score `ckIndex` by a fair discipline
-(SFQ/stride virtual time, or DRR) instead of by head timestamp, advancing the
-per-key state inside the Lua that maintains `ckIndex`. Both spikes agree on the
-discipline; this one shows it works at the real seam. Next step past the spike is
-a design for holding per-key virtual-time state in Redis and advancing it in the
-enqueue/dequeue Lua, plus the multi-shard/multi-consumer story neither spike
-covers.
+Score `ckIndex` by a fair discipline (SFQ/stride virtual time, or DRR) instead of
+by head timestamp. Both spikes agree on the discipline and this one shows the
+ordering fix works through the real dequeue path at `maxCount = 1`. The design
+spike past this needs to: advance per-key virtual-time state inside the batched
+CK-dequeue Lua (the `maxCount > 1` path this spike did not exercise), hold that
+state in Redis for the multi-consumer case, and address the per-CK
+concurrency-limit multiplication that is the other half of #2617.

internal-packages/run-engine/src/run-queue/fairness-spike-ck/harness/ckScenarios.ts

@@ -4,31 +4,44 @@ import type { WorkloadConfig } from "../../fairness-spike/harness/workload.js";
  * Concurrency-key scenarios. Each "tenant" in the workload is a concurrency key
  * under one shared base queue. All keys are equal weight (concurrency keys carry
  * no configured weight in production). Seed is set per-run by the bench.
+ *
+ * Every scenario gives keys genuinely divergent head ages (a bulk key with a
+ * same-time backlog keeps an old head; other keys arrive via poisson so their
+ * heads are distinct and later). This is deliberate: if all runs shared one
+ * enqueue timestamp the ckIndex scores would tie and the real Lua's
+ * ZRANGEBYSCORE would fall back to a lexicographic member-name tie-break, which
+ * would make the baseline look starved for reasons that have nothing to do with
+ * age order. We want the baseline to exercise the real age dynamic #2617
+ * describes.
  */
 export const CK_SCENARIOS: Record<string, Omit<WorkloadConfig, "seed">> = {
-  // The direct #2617 reproduction: one key with a big backlog vs many light keys.
+  // #2617 classic: one key fires a big backlog at once (its head stays old), four
+  // other keys trickle in. Age-order serves the old-headed backlog to exhaustion
+  // and starves the others.
   ckSkew: {
     envConcurrencyLimit: 4,
     tenants: [
       { tenantId: "heavy", runCount: 240, holdMsMean: 25 },
-      { tenantId: "light-1", runCount: 15, holdMsMean: 25 },
-      { tenantId: "light-2", runCount: 15, holdMsMean: 25 },
-      { tenantId: "light-3", runCount: 15, holdMsMean: 25 },
-      { tenantId: "light-4", runCount: 15, holdMsMean: 25 },
+      { tenantId: "light-1", runCount: 15, arrival: "poisson", ratePerSec: 10, holdMsMean: 25 },
+      { tenantId: "light-2", runCount: 15, arrival: "poisson", ratePerSec: 10, holdMsMean: 25 },
+      { tenantId: "light-3", runCount: 15, arrival: "poisson", ratePerSec: 10, holdMsMean: 25 },
+      { tenantId: "light-4", runCount: 15, arrival: "poisson", ratePerSec: 10, holdMsMean: 25 },
     ],
   },
 
+  // Symmetric keys, all arriving via poisson at the same rate. Baseline should be
+  // roughly fair here (no key is systematically older), which is the sanity check.
   ckBalanced: {
     envConcurrencyLimit: 4,
     tenants: [
-      { tenantId: "k-a", runCount: 60, holdMsMean: 25 },
-      { tenantId: "k-b", runCount: 60, holdMsMean: 25 },
-      { tenantId: "k-c", runCount: 60, holdMsMean: 25 },
-      { tenantId: "k-d", runCount: 60, holdMsMean: 25 },
+      { tenantId: "k-a", runCount: 60, arrival: "poisson", ratePerSec: 20, holdMsMean: 25 },
+      { tenantId: "k-b", runCount: 60, arrival: "poisson", ratePerSec: 20, holdMsMean: 25 },
+      { tenantId: "k-c", runCount: 60, arrival: "poisson", ratePerSec: 20, holdMsMean: 25 },
+      { tenantId: "k-d", runCount: 60, arrival: "poisson", ratePerSec: 20, holdMsMean: 25 },
     ],
   },
 
-  // A bulk key plus keys whose runs trickle in, to exercise wait and CoDel.
+  // A bulk backlog plus two keys trickling in slowly, to exercise wait and CoDel.
   ckTrickle: {
     envConcurrencyLimit: 4,
     tenants: [