Skip to content

Measure overlay-base database size at the clear cleanup level#3979

Open
henrymercer wants to merge 3 commits into
mainfrom
henrymercer/overlay-db-cleanup-size-telemetry
Open

Measure overlay-base database size at the clear cleanup level#3979
henrymercer wants to merge 3 commits into
mainfrom
henrymercer/overlay-db-cleanup-size-telemetry

Conversation

@henrymercer

Copy link
Copy Markdown
Contributor

This lets us compare the storage cost of overlay-base and trimmed databases for the same commit.

This only happens when the feature flag for uploading overlay DBs to the GitHub API is enabled (currently staff only) and when running on the default branch. There is some performance penalty to taking this measurement, but since it's on the default branch, this seems justifiable for the time period that we are rolling out uploading overlay DBs to the GitHub API. We could also just compare the zipped DB size with and without the feature flag, but this change has the benefit of giving us the numbers on the same commit, which improves data quality.

Risk assessment

For internal use only. Please select the risk level of this change:

  • Low risk: Changes are fully under feature flags, or have been fully tested and validated in pre-production environments and are highly observable, or are documentation or test only.

Which use cases does this change impact?

Workflow types:

  • Advanced setup - Impacts users who have custom CodeQL workflows.
  • Managed - Impacts users with dynamic workflows (Default Setup, Code Quality, ...).

Products:

  • Code Scanning - The changes impact analyses when analysis-kinds: code-scanning.

Environments:

  • Dotcom - Impacts CodeQL workflows on github.com and/or GitHub Enterprise Cloud with Data Residency.

How did/will you validate this change?

  • Unit tests - I am depending on unit test coverage (i.e. tests in .test.ts files).

If something goes wrong after this change is released, what are the mitigation and rollback strategies?

  • Feature flags - All new or changed code paths can be fully disabled with corresponding feature flags.

How will you know if something goes wrong after this change is released?

  • Telemetry - I rely on existing telemetry or have made changes to the telemetry.
    • Dashboards - I will watch relevant dashboards for issues after the release. Consider whether this requires this change to be released at a particular time rather than as part of a regular release.

Are there any special considerations for merging or releasing this change?

  • No special considerations - This change can be merged at any time.

Merge / deployment checklist

  • Confirm this change is backwards compatible with existing workflows.
  • Consider adding a changelog entry for this change.
  • Confirm the readme and docs have been updated if necessary.

This lets us compare the storage cost of overlay-base and trimmed databases for the same commit.
@github-actions github-actions Bot added the size/M Should be of average difficulty to review label Jun 26, 2026
@henrymercer henrymercer requested a review from Copilot June 26, 2026 16:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

  • Copilot's review of this pull request may be incomplete because some of the changed files are excluded by your Copilot content exclusion settings. See Excluding content from Copilot for details.

Pull request overview

This PR adds telemetry to measure what an overlay-base database would have compressed to if it had been cleaned with CleanupLevel.Clear (instead of CleanupLevel.Overlay), enabling apples-to-apples storage comparisons on the same commit. The measurement runs only when overlay-base uploads are enabled (feature-flagged) and on the default branch, and is skipped in debug mode.

Changes:

  • Extend DatabaseUploadResult with optional fields to record the clear-cleanup zipped size and the time spent measuring it.
  • After overlay-base uploads complete, perform a clear cleanup and re-bundle each language to record the “clear” zipped size (non-fatal, telemetry-only).
  • Add unit tests covering the overlay-base measurement path and the skip conditions (regular uploads, debug mode).
Show a summary per file
File Description
src/database-upload.ts Adds post-upload clear-cleanup size measurement and new telemetry fields for overlay-base runs.
src/database-upload.test.ts Adds tests validating measurement behavior and skip conditions.
lib/entry-points.js Generated output (excluded from diff by policy); expected to track corresponding TypeScript changes.

Review details

Files excluded by content exclusion policy (1)
  • lib/entry-points.js
  • Files reviewed: 2/3 changed files
  • Comments generated: 2
  • Review effort level: Low

Comment thread src/database-upload.ts Outdated
Comment thread src/database-upload.ts Outdated
@henrymercer henrymercer marked this pull request as ready for review June 26, 2026 17:43
@henrymercer henrymercer requested a review from a team as a code owner June 26, 2026 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/M Should be of average difficulty to review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants