perf(core): cache upstream session summaries

[ChanX21]

Summary

Caches successful upstream session summaries in formatUpstreamSessions() to avoid repeated LLM summarization for identical large memory
context.

This helps memory-dependent evaluator flows such as plant-to-trigger checks reuse the same formatted upstream session context without
changing prompt behavior.

Closes #185

Validation

• node --import tsx/esm --test core/tests/summarizeSessionContext.test.ts
• npm run typecheck --workspace=core
• npm test --workspace=core
• pre-commit hook passed

Summary by CodeRabbit

• New Features
  • Added caching for repeated session summarization results, helping return the same summarized context faster when the input and options haven’t changed.
• Bug Fixes
  • Improved consistency by reusing previously generated summaries instead of regenerating identical output multiple times.
• Tests
  • Added coverage to verify repeated summarization calls with the same input only trigger one upstream request.

[coderabbitai]

Walkthrough

Adds an in-memory, per-model bounded cache for formatted upstream session summaries in summarizeSessionContext.ts, keyed by transcript text, maxChars, labelStyle, and sectionHeader, with capped eviction. Includes a regression test confirming repeated identical calls reuse the cache and invoke the summarization endpoint only once.

Changes

Upstream Session Summary Caching

Layer / File(s)	Summary
Cache data structures and helpers core/src/lib/summarizeSessionContext.ts	Adds a MAX_SUMMARY_CACHE_ENTRIES constant, a per-model WeakMap cache (summaryCacheByModel), and helper functions (getSummaryCache, summaryCacheKey, setCachedSummary) to compute cache keys and enforce bounded eviction of the oldest entry.
Cache lookup and population in formatUpstreamSessions core/src/lib/summarizeSessionContext.ts	formatUpstreamSessions checks the per-model cache before invoking the LLM summarizer and returns cached output on a hit; after generating a summary it formats the result and stores it in the cache. Fallback behavior when model is undefined is unchanged.
Regression test for cache reuse core/tests/summarizeSessionContext.test.ts	Adds a test suite using an in-process HTTP server simulating /v1/chat/completions, a deterministic longSession() builder, and a test asserting two identical calls to formatUpstreamSessions return equal output while the summarization endpoint is called only once.

Estimated code review effort: 2 (Simple) | ~15 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant formatUpstreamSessions
  participant SummaryCache
  participant LanguageModel

  Caller->>formatUpstreamSessions: call with transcript, maxChars, labelStyle, sectionHeader
  formatUpstreamSessions->>SummaryCache: lookup by cache key
  alt cache hit
    SummaryCache-->>formatUpstreamSessions: cached formatted summary
    formatUpstreamSessions-->>Caller: return cached summary
  else cache miss
    formatUpstreamSessions->>LanguageModel: request summarization
    LanguageModel-->>formatUpstreamSessions: summary text
    formatUpstreamSessions->>SummaryCache: store formatted summary
    formatUpstreamSessions-->>Caller: return formatted summary
  end

Loading

Related issues: #185

Suggested labels: performance, tests

Suggested reviewers: None identified

Poem:
A rabbit stores what once it read,
No need to ask the same thing twice instead,
The cache holds tight, per model, bounded, neat,
And summarizer calls grow light on their feet,
Hop once, remember — that's the trick indeed. 🐇

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description lacks the required Problem, Solution, Changes, Issue, and How to test sections from the template.	Rewrite it using the template headings and briefly fill each section, including the issue reference and verification steps.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: caching upstream session summaries.
Linked Issues check	✅ Passed	The cache is per-model, bounded, keyed correctly, skips fallback output, and includes a regression test for identical summaries.
Out of Scope Changes check	✅ Passed	The changes stay focused on session-summary caching and its test coverage without unrelated scope creep.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (2)

core/tests/summarizeSessionContext.test.ts (1)

82-102: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Missing regression test for "no caching of failed-summarization fallback".

This test verifies cache reuse on success, but the issue also calls out that fallback output from failed summarization must never be cached. Consider adding a case where the mock endpoint returns an error/non-200 for the first call and a distinct successful response for a retried call with the same key, asserting the second call still hits the endpoint (i.e., the failure wasn't cached).

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@core/tests/summarizeSessionContext.test.ts` around lines 82 - 102, Add a
regression test in summarizeSessionContext.test.ts around formatUpstreamSessions
that covers failed summarization fallback not being cached: use the existing
createModel and srv/mock endpoint setup, make the first summarize call return a
non-200/error path, then retry with the same sessions and cache key but a
successful response, and assert the second call still reaches the endpoint and
does not reuse the failed fallback. Reuse the existing helpers like
longSession(), createModel(), formatUpstreamSessions(), and the
srv.summarizeCalls counter to verify only successful results are cached.

core/src/lib/summarizeSessionContext.ts (1)

92-153: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low value

Cache wiring correctly satisfies the key-composition and no-cache-on-failure requirements.

Cache key includes fullText, maxChars, labelStyle, and sectionHeader The per-model cache used by summarizeSessionContext must align with the fact that the LLM output is determined by the constructed system/user prompts and model.model, and caching is skipped entirely on the catch path (line 150-152), so fallback truncation output is never persisted.

One gap: concurrent identical calls (not just sequential) aren't deduplicated — two in-flight calls with the same key will both hit the LLM before either populates the cache. Given the current call sites appear sequential (awaited per evaluator flow), this is likely low priority, but worth a comment if concurrent invocation becomes common.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@core/src/lib/summarizeSessionContext.ts` around lines 92 - 153, The cache in
formatUpstreamSessions only stores completed summaries, so concurrent identical
requests can still trigger duplicate LLM calls before the first result is
cached. Add in-flight deduplication around the existing
getSummaryCache/summaryCacheKey path in summarizeSessionContext so the same
cache key shares one pending generateText promise, while keeping the current
truncateFallback and catch behavior unchanged.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@core/src/lib/summarizeSessionContext.ts`:
- Around line 92-153: The cache in formatUpstreamSessions only stores completed
summaries, so concurrent identical requests can still trigger duplicate LLM
calls before the first result is cached. Add in-flight deduplication around the
existing getSummaryCache/summaryCacheKey path in summarizeSessionContext so the
same cache key shares one pending generateText promise, while keeping the
current truncateFallback and catch behavior unchanged.

In `@core/tests/summarizeSessionContext.test.ts`:
- Around line 82-102: Add a regression test in summarizeSessionContext.test.ts
around formatUpstreamSessions that covers failed summarization fallback not
being cached: use the existing createModel and srv/mock endpoint setup, make the
first summarize call return a non-200/error path, then retry with the same
sessions and cache key but a successful response, and assert the second call
still reaches the endpoint and does not reuse the failed fallback. Reuse the
existing helpers like longSession(), createModel(), formatUpstreamSessions(),
and the srv.summarizeCalls counter to verify only successful results are cached.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 02c40015-9721-4b9e-8628-3a39e753865a

📥 Commits

Reviewing files that changed from the base of the PR and between 6b2e2b4 and 98dc4d7.

📒 Files selected for processing (2)

• core/src/lib/summarizeSessionContext.ts
• core/tests/summarizeSessionContext.test.ts