Skip to content

Scope orphaned-event cleanup to recent ingest window to eliminate 8-hour ES CPU spikes#2279

Open
niemyjski with Copilot wants to merge 8 commits into
mainfrom
copilot/investigate-cluster-cpu-spikes
Open

Scope orphaned-event cleanup to recent ingest window to eliminate 8-hour ES CPU spikes#2279
niemyjski with Copilot wants to merge 8 commits into
mainfrom
copilot/investigate-cluster-cpu-spikes

Conversation

Copilot AI commented May 31, 2026

Copy link
Copy Markdown
Contributor

What changed

  • Limits orphaned-event stack, project, and organization scans to events created within the last three days.
  • Applies the same cutoff to cardinality queries, partitioned terms queries, and delete-by-query operations.
  • Captures one cutoff per job run so all three cleanup phases use an atomic boundary.
  • Uses the current Elastic.Clients.Elasticsearch and repository architecture from main.
  • Adds regression coverage for stack, project, and organization orphans at and immediately before the inclusive cutoff.

Why

The cleanup job runs every eight hours. Scanning all event history caused unnecessary Elasticsearch CPU spikes even though actionable orphan cleanup is limited to the practical recent-ingest window.

Verification

  • dotnet build tests/Exceptionless.Tests/Exceptionless.Tests.csproj --no-restore --nologo --verbosity:minimal
  • Exceptionless.Tests --filter-class Exceptionless.Tests.Jobs.CleanupDataJobTests: 28 passed
  • Exact lookback-boundary regression: passed
  • Local Aspire dogfood with healthy Elasticsearch 8.19.15, Redis, API, and Jobs resources

Compatibility

No public API, WebSocket, configuration, or frontend behavior changes.

@CLAassistant

CLAassistant commented May 31, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ niemyjski
❌ Copilot
You have signed the CLA already but the status is still pending? Let us recheck it.

Copilot AI linked an issue Jun 1, 2026 that may be closed by this pull request
Copilot AI and others added 2 commits June 1, 2026 00:22
Co-authored-by: niemyjski <1020579+niemyjski@users.noreply.github.com>
Co-authored-by: niemyjski <1020579+niemyjski@users.noreply.github.com>
Copilot AI changed the title [WIP] Investigate cluster CPU spikes during orphan data cleanup Scope orphaned-event cleanup to recent ingest window to eliminate 8-hour ES CPU spikes Jun 1, 2026
Copilot AI requested a review from niemyjski June 1, 2026 00:28
…luster-cpu-spikes

# Conflicts:
#	src/Exceptionless.Core/Jobs/CleanupOrphanedDataJob.cs
#	tests/Exceptionless.Tests/Jobs/CleanupDataJobTests.cs
@niemyjski
niemyjski marked this pull request as ready for review July 10, 2026 19:34

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c9531879f6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/Exceptionless.Core/Jobs/CleanupOrphanedDataJob.cs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b0910ef8ad

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/Exceptionless.Core/Jobs/CleanupOrphanedDataJob.cs
@github-actions

Copy link
Copy Markdown

Code Coverage

Package Line Rate Branch Rate Complexity Health
Exceptionless.AppHost 38% 39% 144
Exceptionless.Core 72% 64% 8886
Exceptionless.Insulation 23% 23% 205
Exceptionless.Web 84% 67% 6841
Summary 76% (21392 / 28169) 65% (9828 / 15190) 16076

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Investigate cluster CPU spikes

3 participants