Reduce shim instance-start polling load with bounded jitter by berndverst · Pull Request #779 · microsoft/durabletask-dotnet

berndverst · 2026-07-24T22:05:28Z

Summary

ShimDurableTaskClient.WaitForInstanceStartAsync previously polled GetOrchestrationStateAsync on a fixed 1-second Task.Delay cadence. Under load with many concurrent waiters, this causes synchronized polling bursts against the backend instead of a smooth request rate (#776).

Changes

The first status check remains immediate (no delay), preserving prompt observation of quick-starting orchestrations.
Subsequent delays are computed by a new internal static ComputeNextPollingDelay(bool isInitialDelay) helper:
- Only the very first delay of a given WaitForInstanceStartAsync call is a randomized phase offset, uniformly distributed in [0s, 1s). This is a one-time cost per call, not repeated per iteration.
- Every subsequent delay is the fixed historical PollingInterval (1 second), completely unjittered.
This desynchronizes concurrent callers (avoiding synchronized polling bursts against the backend) via the one-time random phase, without inflating steady-state polling volume — after the first delay, the cadence exactly matches the historical fixed 1-second rate forever, so long-run polling frequency never exceeds the historical rate.
The worst-case detection latency for any single status transition still never exceeds the historical 1 second.
Not-found behavior is unchanged: if the instance never existed, InvalidOperationException is thrown immediately with no polling delay.
Cancellation behavior is unchanged: the CancellationToken is still honored on every status check and on Task.Delay.

Revision history on this PR

Original version: exponential backoff (1s → 5s cap, 1.5x multiplier) + ±20% jitter. Reduced steady-state load for long-pending instances, but review found it could delay detection of a newly-started orchestration by up to ~6s in the worst case — a regression against the historical 1s guarantee.
First revision: removed growth; jitter applied only downward from a fixed 1s base (range [0.8s, 1.0s]), guaranteeing the 1s ceiling. Review found this increases steady-state polling volume by ~11% (mean 0.9s vs. historical 1.0s), undermining the load-reduction goal, and flagged the Stopwatch-based test as CI-flaky (Task.Delay has no guaranteed upper bound).
Third revision: one-time randomized initial phase offset ([0s, 1s)), then fixed unjittered 1s intervals thereafter. Preserves the 1s worst-case detection latency ceiling, desynchronizes concurrent callers, and keeps steady-state polling volume exactly at the historical rate (no increase, no decrease). This design is unchanged as of the current revision.
Fourth revision: no production behavior change. Review found WaitForInstanceStart_CancelledDuringPollingDelay_ThrowsTaskCanceledException used wall-clock cts.CancelAfter(50ms), which races against when the loop actually enters Task.Delay — so the test could still "pass" via an unrelated code path (e.g. a later poll observing an already-cancelled token) without proving the delay itself honors cancellation. Fixed by adding an internal virtual DelayAsync(TimeSpan, CancellationToken) seam (defaults to Task.Delay) that a test-only subclass used to signal, via a TaskCompletionSource, the exact moment the real delay call had been made.
Current version (this revision): no production behavior change; test-only fix. Review found the fourth revision's test double still let the real Task.Delay(delay, cancellation) run, so a regression passing the wrong (or no) token into DelayAsync could still "pass": GetInstancesAsync's own cancellation.ThrowIfCancellationRequested() check would throw on the next poll attempt before the mock is invoked again, satisfying the Times.Once assertion even though the delay itself never honored cancellation. Fixed by replacing the fake delay with one that never completes on its own (no real timer) and completes — via cancellation — only when the exact CancellationToken instance passed by the caller is cancelled, directly proving cancellation of the caller's token (not some other code path) ends the wait. Verified by temporarily reintroducing the regression (CancellationToken.None at the call site) and confirming the test now fails deterministically instead of passing vacuously; reverted after confirming.

Testing

Focused tests in ShimDurableTaskClientTests.cs:

WaitForInstanceStart_MultiplePendingPolls_EventuallyReturnsTerminalMetadata — multiple polling iterations still converge on the terminal state.
ComputeNextPollingDelay_InitialDelay_NeverExceedsHistoricalOneSecondCadence — deterministic: 1000 samples of the initial-phase delay, asserting each is in [0s, 1s).
ComputeNextPollingDelay_SteadyState_ReturnsFixedHistoricalIntervalWithNoJitter — deterministic: every subsequent delay equals exactly 1 second (no jitter, no growth), proving steady-state volume can't regress in either direction. This directly tests the delay policy rather than relying on wall-clock timing, avoiding the CI flakiness of the earlier Stopwatch-based approach.
WaitForInstanceStart_InstanceNotFound_ThrowsImmediatelyWithoutPolling — not-found short-circuits without any polling delay.
WaitForInstanceStart_CancelledDuringPollingDelay_ThrowsTaskCanceledException — fully deterministic (no wall-clock coordination, no real timer): a test-only DelayObservingShimDurableTaskClient subclass overrides the internal DelayAsync seam with a fake delay that signals as soon as it is entered, then never completes on its own — it completes, via cancellation, only when the exact CancellationToken instance passed by the caller is cancelled. The test cancels only after that signal, and asserts GetOrchestrationStateAsync was called Times.Once. This proves cancelling the caller's token — not some unrelated code path such as a later poll's own cancellation check — is what ends the wait. Verified by temporarily reintroducing a "wrong token" regression at the call site and confirming the test then fails deterministically instead of passing vacuously.

The pre-existing WaitForInstanceStart test (asserting exactly 2 GetOrchestrationStateAsync calls) continues to pass unchanged.

Ran the full Client.OrchestrationServiceClientShim.Tests suite: all 76 tests pass, 0 failures (~2s runtime). Re-ran the cancellation test 5x in isolation to confirm it is not flaky (all passed, ~160ms each).

Breaking Change

No breaking change — WaitForInstanceStartAsync's public signature and behavior are unchanged. ComputeNextPollingDelay and the internal DelayAsync test seam are internal (not public) purely to allow direct, deterministic unit testing; they and the polling constants remain implementation details of the internal ShimDurableTaskClient class. No public configuration was added.

Unrelated CI stabilization

CI on this PR was blocked by a reproducible timeout in WorkItemStreamConsumerTests.PerItem_HeartbeatReset_KeepsTimerAlive (test/Worker/Grpc.Tests), unrelated to the shim polling change above (same flake reproduces on main). Root cause: the test slept a fixed 150ms before writing the first channel item, racing against the consumer's 500ms silent-disconnect timer, which is armed as soon as ConsumeAsync starts. Under CI scheduling pressure that sleep could exceed the timer window, causing the test to hang.

Fix (test-only, no production behavior change): the first item is now written immediately with no a-priori sleep, removing the race. To preserve a strong regression signal for a missing per-item timer reset, the test now sends several items in sequence with each gap measured from the previous item's actual processing (via a semaphore signal, not a wall-clock guess): each individual gap (150ms) stays comfortably below the 500ms timeout, but the gaps' sum (600ms) comfortably exceeds it, so an implementation that only arms the timer once at loop start (never re-arming per item) fails this test well before the last item is sent.

Verified by temporarily removing the per-item ArmSilentDisconnectTimer() call in production and confirming the test failed deterministically; reverted (zero production diff). Ran the test 10x standalone (stable) and the full Worker.Grpc.Tests suite (137/137 pass).

Follow-up: made the CI-stabilization test fully deterministic (commit `8779f98`)

A further review found the previous stabilization (commit c8b66aa) still had residual, if much smaller, wall-clock risk: it measured real 150ms per-item gaps against the real 500ms silent-disconnect timeout, so a scheduler-delayed continuation between "item processed" and "next write" could in theory still inflate a gap past the timeout under CI pressure.

Fix: added a test-only observability seam to WorkItemStreamConsumer.ConsumeAsync -- an optional onSilentDisconnectTimerArmed callback (default null, zero production behavior change, purely additive trailing parameter) invoked every time the silent-disconnect timer is armed/re-armed. PerItem_HeartbeatReset_KeepsTimerAlive now records the exact interleaving of "armed" and "item" events and asserts the structural invariant directly (one arm before the loop, one re-arm immediately before each item), with no wall-clock timing at all. The test now runs in ~20ms.

Verified red/green: temporarily disabled the per-item re-arm in production, confirmed the test failed deterministically (armed count 1 instead of 6), reverted (diff confirms only the seam addition remains). Ran the test 15x standalone (all pass), full Worker.Grpc.Tests (137/137), and full Client.OrchestrationServiceClientShim.Tests (76/76, confirms the shim polling work is unaffected).

Fixes #776

WaitForInstanceStartAsync previously polled GetOrchestrationStateAsync on a fixed 1-second Task.Delay cadence. Under load, many concurrent waiters end up synchronized on the same polling tick, causing bursty spikes of backend requests instead of a smooth request rate. This changes the wait loop to: - Keep the very first status check immediate (no delay), preserving prompt observation of quick-starting orchestrations. - Compute each subsequent delay via a new ComputeNextPollingDelay helper: starts at the historical 1s interval, grows by a 1.5x multiplier per attempt (capped), up to a 5s maximum. - Apply +/-20% jitter (via a lock-guarded, securely-seeded Random in the new PollingJitter helper) so concurrent callers desynchronize instead of polling in lockstep. Cancellation, not-found (throws immediately, no delay), and terminal-state return behavior are unchanged. No public API surface was touched -- WaitForInstanceStartAsync's signature is unchanged and all new members are private implementation details, so this is not a breaking change. Added focused tests covering multi-iteration polling convergence, not-found short-circuiting without polling delay, and cancellation during the backoff delay. Fixes #776 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8aa98c91-6050-44ef-b236-dd1cf43fabac

Copilot

Pull request overview

This PR reduces backend load caused by synchronized polling in ShimDurableTaskClient.WaitForInstanceStartAsync by introducing a gradual backoff policy with per-attempt jitter, while preserving immediate first-check behavior, not-found semantics, and cancellation support.

Changes:

Replaced fixed 1-second polling with a bounded exponential backoff (min 1s, max 5s) plus ±20% jitter to desynchronize concurrent callers.
Added private helpers (ComputeNextPollingDelay and PollingJitter) to centralize delay/backoff/jitter behavior.
Added unit tests covering multi-poll convergence, not-found short-circuit, and cancellation during the delay.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
src/Client/OrchestrationServiceClientShim/ShimDurableTaskClient.cs	Adds jittered backoff delay logic to smooth polling load in `WaitForInstanceStartAsync`.
test/Client/OrchestrationServiceClientShim.Tests/ShimDurableTaskClientTests.cs	Adds targeted tests for multiple pending polls, not-found behavior, and cancellation during backoff.

+    }
+
+    [Fact]
+    public async Task WaitForInstanceStart_CancelledDuringBackoffDelay_ThrowsTaskCanceledException()


…ence The prior jittered exponential-backoff design (base 1s, cap 5s, +/-20% jitter) could delay detection of a newly-started orchestration by up to ~6s, regressing the historical 1s worst-case detection latency that the original issue explicitly required to be preserved. Replace growth-based backoff with a fixed 1s PollingInterval that jitter can only ever reduce, never exceed. This still desynchronizes concurrent callers (avoiding synchronized polling bursts) while guaranteeing the delay is always in [0.8s, 1.0s] -- so worst-case detection latency never regresses past the historical 1s cadence. ComputeNextPollingDelay is changed from private static to internal static (not a public API surface change) so it can be exercised directly and deterministically by a new unit test enforcing the max-latency bound. Also adds a Stopwatch-based integration test covering repeated pending observations before a terminal transition, asserting the total wait stays well under the historical cadence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8aa98c91-6050-44ef-b236-dd1cf43fabac

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

+    /// A delay in the range [<see cref="PollingInterval"/> * (1 - <see cref="PollingJitterFactor"/>),
+    /// <see cref="PollingInterval"/>]. Jitter only ever reduces the delay, so the returned value never
+    /// exceeds <see cref="PollingInterval"/> -- preserving the historical worst-case detection latency
+    /// for <see cref="WaitForInstanceStartAsync"/> -- while still desynchronizing concurrent callers.


+            // Poll with a delay that is jittered *downward* from the historical 1-second cadence.
+            // This desynchronizes concurrent waiters (avoiding synchronized polling bursts against the
+            // backend) while guaranteeing the worst-case detection latency for a status transition
+            // never exceeds the historical 1-second cadence -- preserving prompt-start observation.
+            TimeSpan delay = ComputeNextPollingDelay();
+            await Task.Delay(delay, cancellation);


+    }
+
+    [Fact]
+    public async Task WaitForInstanceStart_CancelledDuringPollingDelay_ThrowsTaskCanceledException()


…uction Terra re-review flagged that the previous downward-only jitter design (range [0.8s, 1.0s], mean 0.9s) increases steady-state polling volume by ~11% versus the historical fixed 1s cadence, undermining the load-reduction goal of #776. It also flagged that the Stopwatch-based upper-bound integration test was CI-flaky, since Task.Delay only guarantees a lower bound on elapsed time, not an upper one. Replace the per-iteration downward jitter with a one-time randomized initial phase offset: only the first delay of a given WaitForInstanceStartAsync call is randomized (uniformly in [0, 1s)); every delay after that is the fixed historical 1-second interval, completely unjittered. This still desynchronizes concurrent callers (avoiding synchronized polling bursts) via the one-time phase difference, but no longer inflates steady-state polling volume, since subsequent delays exactly match the historical cadence forever. The worst-case detection latency for any single status transition still never exceeds the historical 1 second. Replace the flaky Stopwatch-based test with two deterministic unit tests against the (internal) delay-computation policy directly: - ComputeNextPollingDelay_InitialDelay_NeverExceedsHistoricalOneSecondCadence asserts 1000 samples of the initial-phase delay are always in [0, 1s). - ComputeNextPollingDelay_SteadyState_ReturnsFixedHistoricalIntervalWithNoJitter asserts every subsequent delay is exactly 1 second, with no jitter and no growth -- proving steady-state polling volume cannot regress in either direction. No public API change; ComputeNextPollingDelay remains an internal-only member of the internal ShimDurableTaskClient class. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8aa98c91-6050-44ef-b236-dd1cf43fabac

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

+        if (isInitialDelay)
+        {
+            return TimeSpan.FromMilliseconds(PollingInterval.TotalMilliseconds * PollingJitter.NextDouble());
+        }


Replace wall-clock cts.CancelAfter(50ms) coordination in WaitForInstanceStart_CancelledDuringPollingDelay_ThrowsTaskCanceledException with deterministic coordination: an internal virtual DelayAsync seam on ShimDurableTaskClient lets the test observe (via a TaskCompletionSource) the exact moment the real Task.Delay(delay, cancellation) call for the polling loop has been made, and only then cancels the token. The test also asserts GetOrchestrationStateAsync was called exactly once, proving cancellation ended the wait during the delay itself rather than via a subsequent poll happening to observe an already-cancelled token. No production behavior change: DelayAsync defaults to Task.Delay and is internal (not public API surface), so no breaking-change-check is required. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8aa98c91-6050-44ef-b236-dd1cf43fabac

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

src/Client/OrchestrationServiceClientShim/ShimDurableTaskClient.cs:390

ComputeNextPollingDelay uses TimeSpan.FromMilliseconds(double). Because TimeSpan.FromMilliseconds rounds to the nearest tick, values very close to 1s can round up and return exactly PollingInterval, contradicting the documented "[0, PollingInterval)" range and making the "< 1 second" unit test theoretically flaky. Using tick-based truncation avoids this rounding edge case and guarantees the delay is strictly less than PollingInterval.

    internal static TimeSpan ComputeNextPollingDelay(bool isInitialDelay)
    {
        if (isInitialDelay)
        {
            return TimeSpan.FromMilliseconds(PollingInterval.TotalMilliseconds * PollingJitter.NextDouble());
        }

test/Client/OrchestrationServiceClientShim.Tests/ShimDurableTaskClientTests.cs:425

The test name says it throws TaskCanceledException, but the assertion checks for OperationCanceledException. If the intent is to validate the specific exception type produced by Task.Delay cancellation (and match the test name), assert TaskCanceledException; otherwise, rename the test to avoid the mismatch.

        completedTask.Should().Be(waitTask, "the wait should be cancelled during the polling delay, not time out");
        Func<Task> act = () => waitTask;
        await act.Should().ThrowAsync<OperationCanceledException>();
        this.orchestrationClient.Verify(

+            .ReturnsAsync([pending]);
+
+        using CancellationTokenSource cts = new();
+        DelayObservingShimDurableTaskClient client = new(


Terra's fourth review found the previous fix still insufficient: the test double called the real Task.Delay(delay, cancellation) and let it run, so a regression that passed the wrong (or no) token into DelayAsync could still 'pass' the test -- GetInstancesAsync's own cancellation.ThrowIfCancellationRequested() check would throw on the next poll attempt before the mock is invoked again, satisfying the Times.Once assertion even though the delay itself never honored cancellation. Replace the fake delay with one that never completes on its own (no real timer) and completes -- via cancellation -- only when the exact CancellationToken instance passed by the caller is cancelled. This directly proves that cancelling the caller's token is what ends the wait, independent of any other cancellation check in the poll loop. Verified by temporarily reintroducing the regression (passing CancellationToken.None into DelayAsync at the call site) and confirming the test now fails deterministically (times out at the 10s safety net) instead of passing vacuously; reverted after confirming. No production behavior change -- test-only fix. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8aa98c91-6050-44ef-b236-dd1cf43fabac

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

test/Client/OrchestrationServiceClientShim.Tests/ShimDurableTaskClientTests.cs:387

The test name says it throws TaskCanceledException, but the assertion accepts any OperationCanceledException. Either tighten the assertion to TaskCanceledException or rename the test to match what it asserts.

    public async Task WaitForInstanceStart_CancelledDuringPollingDelay_ThrowsTaskCanceledException()

+        // act
+        OrchestrationMetadata metadata = await this.client.WaitForInstanceStartAsync(
+            instance.InstanceId, false, default);
+


…g flake This test is unrelated to the shim polling work in this PR, but was blocking CI on this branch with a reproducible timeout under scheduling pressure. Root cause: the test slept 150ms (a fixed wall-clock delay) before writing the first channel item, racing against the consumer's 500ms silent-disconnect timer which is armed as soon as ConsumeAsync starts. Under CI scheduling pressure that 150ms sleep could take long enough to let the timer fire before the first item was ever written, causing the test to hang waiting for a signal that would never arrive. Fix (test-only, no production behavior change): write the first item immediately with no a-priori sleep, eliminating that race entirely. To keep a strong regression signal for a missing per-item timer reset, the test now sends several items in sequence with gaps measured from each item's actual processing (via a semaphore signal, not a wall-clock guess): each individual gap (150ms) is comfortably below the 500ms timeout, but their sum (600ms) comfortably exceeds it -- so a regression that only arms the timer once at loop start would fail this test well before the last item is sent. Verified the fix still catches the regression: temporarily removed the per-item ArmSilentDisconnectTimer() call in WorkItemStreamConsumer.cs and confirmed the test failed deterministically; reverted (zero production diff). Ran the test 10x standalone (stable, ~660ms each) and the full Worker.Grpc.Tests suite (137/137 pass). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8aa98c91-6050-44ef-b236-dd1cf43fabac

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

test/Client/OrchestrationServiceClientShim.Tests/ShimDurableTaskClientTests.cs:426

Test name says TaskCanceledException but the assertion currently accepts any OperationCanceledException. Since the test coordinates cancellation specifically during the polling delay, assert TaskCanceledException (or rename the test) to keep intent and verification consistent.

        completedTask.Should().Be(waitTask, "the wait should be cancelled during the polling delay, not time out");
        Func<Task> act = () => waitTask;
        await act.Should().ThrowAsync<OperationCanceledException>();

+        internal override Task DelayAsync(TimeSpan delay, CancellationToken cancellation)
+        {
+            TaskCompletionSource pending = new(TaskCreationOptions.RunContinuationsAsynchronously);
+            cancellation.Register(() => pending.TrySetCanceled(cancellation));
+            this.delayEntered.TrySetResult();
+            return pending.Task;
+        }


+        TimeSpan perItemGap = TimeSpan.FromMilliseconds(150);
+        const int itemCount = 5; // 4 gaps * 150ms = 600ms > 500ms timeout: proves reset is required.
+
+        SemaphoreSlim itemProcessed = new(0);


…stic The previous fix for the WorkItemStreamConsumerTests CI flake still relied on real per-item delays (150ms each, comfortably under the 500ms timeout) racing against the real silent-disconnect timer. Under CI scheduling pressure a delayed continuation between the "item processed" signal and the next write could still inflate an intended-short gap past the timeout, even though production behavior was correct. Add a test-only observability seam to WorkItemStreamConsumer.ConsumeAsync: an optional onSilentDisconnectTimerArmed callback invoked synchronously every time the silent-disconnect timer is (re-)armed (once before the read loop, once per item). It is null in production (default parameter, purely additive, zero behavior change when unused) and only appended as a trailing optional parameter, so the sole production call site is unaffected. Rewrite PerItem_HeartbeatReset_KeepsTimerAlive to use this seam instead of wall-clock timing: it records the exact interleaving of "armed" and "item" events and asserts the structural invariant directly (one arm before the loop, one re-arm immediately before each item is dispatched) rather than inferring it from elapsed real time. The test now runs in ~20ms with zero timing dependency. Verified red/green: temporarily commented out the per-item ArmSilentDisconnectTimer() call in production, confirmed the test failed deterministically (armed count 1 instead of 6), then reverted (git diff confirms only the intended seam addition remains). Ran the fixed test 15x standalone (all pass, ~20ms each), the full Worker.Grpc.Tests suite (137/137), and the full Client.OrchestrationServiceClientShim.Tests suite (76/76, confirms the core PR content is unaffected). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8aa98c91-6050-44ef-b236-dd1cf43fabac

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

test/Client/OrchestrationServiceClientShim.Tests/ShimDurableTaskClientTests.cs:427

The test name says it throws TaskCanceledException, but the assertion currently allows any OperationCanceledException. Since this test is specifically coordinating cancellation during the polling delay, assert TaskCanceledException exactly to ensure the cancellation is coming from the awaited delay task (and to match the test name).

        completedTask.Should().Be(waitTask, "the wait should be cancelled during the polling delay, not time out");
        Func<Task> act = () => waitTask;
        await act.Should().ThrowAsync<OperationCanceledException>();
        this.orchestrationClient.Verify(

src/Client/OrchestrationServiceClientShim/ShimDurableTaskClient.cs:389

ComputeNextPollingDelay(true) uses TimeSpan.FromMilliseconds(PollingInterval.TotalMilliseconds * NextDouble()). Because TimeSpan.FromMilliseconds rounds to the nearest tick, values very close to 1000ms can round up to exactly PollingInterval (1s), violating the intended [0, PollingInterval) bound and making the corresponding unit test potentially flaky. Compute the jitter using ticks (truncation) to guarantee the result is always < PollingInterval.

            return TimeSpan.FromMilliseconds(PollingInterval.TotalMilliseconds * PollingJitter.NextDouble());

This branch predates the deterministic CI repair applied on berndverst-reduce-shim-polling-load (commits c8b66aa and 8779f98), so PR #780's "Validate Build" job was failing on a flaky, unrelated baseline test: WorkItemStreamConsumerTests.PerItem_HeartbeatReset_KeepsTimerAlive intermittently reported SilentDisconnect instead of GracefulDrain under CI scheduling pressure. Root cause (from c8b66aa/8779f98): the test raced a real per-item wall-clock delay against the real 500ms silent-disconnect timer. Any scheduler delay in a continuation between the "item processed" signal and the next write could inflate an intended-short gap past the timeout, spuriously tripping SilentDisconnect even though production behavior (per-item timer reset) was correct. Fix (test-only, plus one purely-additive optional-parameter observability seam -- no shim polling or other production behavior changes): - WorkItemStreamConsumer.ConsumeAsync gains an optional trailing onSilentDisconnectTimerArmed callback, invoked synchronously every time the silent-disconnect timer is (re-)armed (once before the read loop, once per item). Defaults to null; the sole production call site is unaffected. - PerItem_HeartbeatReset_KeepsTimerAlive now uses this seam to assert the exact "armed"/"item" event interleaving directly, instead of inferring the per-item reset from elapsed real time. Runs in ~20ms with zero timing dependency. Cherry-picked directly from c8b66aa and 8779f98 (verified byte-identical result via `git diff 8779f98 -- <files>` producing no output); this branch's copies of both files were unmodified prior to this transplant, so the cherry-pick applied cleanly with no conflicts. Verified: full Worker.Grpc.Tests suite 147/147 passing; the previously-flaky test run 5x standalone, consistently ~20ms with no flakiness. No changes to NewGuid()/TaskOrchestrationContextWrapper or the round-9 cache fix in this commit. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9b8e1d4-b96d-41ec-adeb-a691aac6fee1

Copilot AI review requested due to automatic review settings July 24, 2026 22:05

Copilot started reviewing on behalf of berndverst July 24, 2026 22:06 View session

Copilot AI reviewed Jul 24, 2026

View reviewed changes

Comment thread test/Client/OrchestrationServiceClientShim.Tests/ShimDurableTaskClientTests.cs Outdated

}

[Fact]

public async Task WaitForInstanceStart_CancelledDuringBackoffDelay_ThrowsTaskCanceledException()

Copilot AI review requested due to automatic review settings July 24, 2026 23:19

berndverst changed the title ~~Reduce shim instance-start polling load with jitter and gradual backoff~~ Reduce shim instance-start polling load with bounded jitter Jul 24, 2026

Copilot started reviewing on behalf of berndverst July 24, 2026 23:20 View session

Copilot AI reviewed Jul 24, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 24, 2026 23:53

Copilot started reviewing on behalf of berndverst July 24, 2026 23:54 View session

Copilot AI reviewed Jul 24, 2026

View reviewed changes

Comment thread src/Client/OrchestrationServiceClientShim/ShimDurableTaskClient.cs

Comment on lines +387 to +390

if (isInitialDelay)

{

return TimeSpan.FromMilliseconds(PollingInterval.TotalMilliseconds * PollingJitter.NextDouble());

}

Copilot AI review requested due to automatic review settings July 25, 2026 00:03

Copilot started reviewing on behalf of berndverst July 25, 2026 00:04 View session

Copilot AI reviewed Jul 25, 2026

View reviewed changes

github-code-quality Bot found potential problems Jul 25, 2026

View reviewed changes

Comment thread test/Client/OrchestrationServiceClientShim.Tests/ShimDurableTaskClientTests.cs

.ReturnsAsync([pending]);

using CancellationTokenSource cts = new();

DelayObservingShimDurableTaskClient client = new(

Copilot AI review requested due to automatic review settings July 25, 2026 00:10

berndverst force-pushed the berndverst-reduce-shim-polling-load branch from bba85fe to 3a74e2d Compare July 25, 2026 00:10

Copilot started reviewing on behalf of berndverst July 25, 2026 00:11 View session

Copilot AI reviewed Jul 25, 2026

View reviewed changes

Comment thread test/Client/OrchestrationServiceClientShim.Tests/ShimDurableTaskClientTests.cs

Comment on lines +328 to +331

// act

OrchestrationMetadata metadata = await this.client.WaitForInstanceStartAsync(

instance.InstanceId, false, default);

Copilot AI review requested due to automatic review settings July 25, 2026 01:13

Copilot started reviewing on behalf of berndverst July 25, 2026 01:14 View session

github-code-quality Bot found potential problems Jul 25, 2026

View reviewed changes

Comment thread test/Worker/Grpc.Tests/WorkItemStreamConsumerTests.cs Fixed

Copilot AI reviewed Jul 25, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 25, 2026 01:26

Copilot started reviewing on behalf of berndverst July 25, 2026 01:27 View session

Copilot AI reviewed Jul 25, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Reduce shim instance-start polling load with bounded jitter#779

Reduce shim instance-start polling load with bounded jitter#779
berndverst wants to merge 7 commits into
mainfrom
berndverst-reduce-shim-polling-load

berndverst commented Jul 24, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

berndverst commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Revision history on this PR

Testing

Breaking Change

Unrelated CI stabilization

Follow-up: made the CI-stabilization test fully deterministic (commit 8779f98)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

berndverst commented Jul 24, 2026 •

edited

Loading

Follow-up: made the CI-stabilization test fully deterministic (commit `8779f98`)