diff --git a/docs/admin/code-hosts/aws-codecommit.mdx b/docs/admin/code-hosts/aws-codecommit.mdx index da3a764c5..d6c91f799 100644 --- a/docs/admin/code-hosts/aws-codecommit.mdx +++ b/docs/admin/code-hosts/aws-codecommit.mdx @@ -37,7 +37,7 @@ AWS CodeCommit connections support the following configuration options, which ar {/* SCHEMA_SYNC_START: admin/code_hosts/aws_codecommit.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-06-17T20:29:01Z */} +{/* Last updated: 2026-07-02T18:22:42Z */} ```json { // REQUIRED: diff --git a/docs/admin/code-hosts/azuredevops.mdx b/docs/admin/code-hosts/azuredevops.mdx index 759a7ec9a..6a9662dd5 100644 --- a/docs/admin/code-hosts/azuredevops.mdx +++ b/docs/admin/code-hosts/azuredevops.mdx @@ -69,7 +69,7 @@ Azure DevOps connections support the following configuration options, which are {/* SCHEMA_SYNC_START: admin/code_hosts/azuredevops.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-06-17T20:29:01Z */} +{/* Last updated: 2026-07-02T18:22:42Z */} ```json // Authentication alternatives: token OR windowsPassword diff --git a/docs/admin/code-hosts/bitbucket-cloud.mdx b/docs/admin/code-hosts/bitbucket-cloud.mdx index 61efa8942..27dec327d 100644 --- a/docs/admin/code-hosts/bitbucket-cloud.mdx +++ b/docs/admin/code-hosts/bitbucket-cloud.mdx @@ -116,7 +116,7 @@ Bitbucket Cloud connections support the following configuration options, which a {/* SCHEMA_SYNC_START: admin/code_hosts/bitbucket_cloud.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-06-17T20:29:01Z */} +{/* Last updated: 2026-07-02T18:22:42Z */} ```json { // The workspace access token to use when authenticating with Bitbucket Cloud. diff --git a/docs/admin/code-hosts/bitbucket-server.mdx b/docs/admin/code-hosts/bitbucket-server.mdx index ec3ec526f..8b98779dc 100644 --- a/docs/admin/code-hosts/bitbucket-server.mdx +++ b/docs/admin/code-hosts/bitbucket-server.mdx @@ -202,7 +202,7 @@ Bitbucket Server / Bitbucket Data Center connections support the following confi {/* SCHEMA_SYNC_START: admin/code_hosts/bitbucket_server.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-06-17T20:29:01Z */} +{/* Last updated: 2026-07-02T18:22:42Z */} ```json // Authentication alternatives: token OR password diff --git a/docs/admin/code-hosts/gerrit.mdx b/docs/admin/code-hosts/gerrit.mdx index 158e2d36c..eea88448a 100644 --- a/docs/admin/code-hosts/gerrit.mdx +++ b/docs/admin/code-hosts/gerrit.mdx @@ -113,7 +113,7 @@ Gerrit connections support the following configuration options, which are specif {/* SCHEMA_SYNC_START: admin/code_hosts/gerrit.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-06-17T20:29:01Z */} +{/* Last updated: 2026-07-02T18:22:42Z */} ```json { // If non-null, enforces Gerrit repository permissions. This requires that there is an item in the [site configuration json](https://sourcegraph.com/docs/admin/config/site_config#auth-providers) `auth.providers` field, of type "gerrit" with the same `url` field as specified in this `GerritConnection`. diff --git a/docs/admin/code-hosts/github.mdx b/docs/admin/code-hosts/github.mdx index 2e5c92390..dc5922663 100644 --- a/docs/admin/code-hosts/github.mdx +++ b/docs/admin/code-hosts/github.mdx @@ -454,7 +454,7 @@ GitHub connections support the following configuration options, which are specif {/* SCHEMA_SYNC_START: admin/code_hosts/github.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-06-17T20:29:01Z */} +{/* Last updated: 2026-07-02T18:22:42Z */} ```json // Authentication alternatives: token OR gitHubAppDetails OR externalAccount OR useRandomExternalAccount diff --git a/docs/admin/code-hosts/gitlab.mdx b/docs/admin/code-hosts/gitlab.mdx index 2f35fee0e..d4599f0b1 100644 --- a/docs/admin/code-hosts/gitlab.mdx +++ b/docs/admin/code-hosts/gitlab.mdx @@ -189,7 +189,7 @@ See [Internal rate limits](/admin/code-hosts/rate-limits#internal-rate-limits). {/* SCHEMA_SYNC_START: admin/code_hosts/gitlab.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-06-17T20:29:01Z */} +{/* Last updated: 2026-07-02T18:22:42Z */} ```json { // If non-null, enforces GitLab repository permissions. This requires that there be an item in the `auth.providers` field of type "gitlab" with the same `url` field as specified in this `GitLabConnection`. diff --git a/docs/admin/code-hosts/gitolite.mdx b/docs/admin/code-hosts/gitolite.mdx index 6c811da9e..0bbb0c988 100644 --- a/docs/admin/code-hosts/gitolite.mdx +++ b/docs/admin/code-hosts/gitolite.mdx @@ -25,7 +25,7 @@ To connect Gitolite to Sourcegraph: {/* SCHEMA_SYNC_START: admin/code_hosts/gitolite.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-06-17T20:29:01Z */} +{/* Last updated: 2026-07-02T18:22:42Z */} ```json { // A list of repositories to never mirror from this Gitolite instance. Supports excluding by exact name ({"name": "foo"}). diff --git a/docs/admin/code-hosts/other.mdx b/docs/admin/code-hosts/other.mdx index af3a0c32b..d378e61fe 100644 --- a/docs/admin/code-hosts/other.mdx +++ b/docs/admin/code-hosts/other.mdx @@ -68,7 +68,7 @@ Repositories must be listed individually: {/* SCHEMA_SYNC_START: admin/code_hosts/other_external_service.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-06-17T20:29:01Z */} +{/* Last updated: 2026-07-02T18:22:42Z */} ```json { // A list of repositories to never mirror by name after applying repositoryPathPattern. Supports excluding by exact name ({"name": "myrepo"}) or regular expression ({"pattern": ".*secret.*"}). diff --git a/docs/admin/code-hosts/phabricator.mdx b/docs/admin/code-hosts/phabricator.mdx index 3ba085b9f..e0b4edac8 100644 --- a/docs/admin/code-hosts/phabricator.mdx +++ b/docs/admin/code-hosts/phabricator.mdx @@ -76,7 +76,7 @@ The Sourcegraph instance's site admin must [update the `corsOrigin` site config {/* SCHEMA_SYNC_START: admin/code_hosts/phabricator.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-06-17T20:29:01Z */} +{/* Last updated: 2026-07-02T18:22:42Z */} ```json { // SSH cipher to use when cloning via SSH. Must be a valid choice from `ssh -Q cipher`. diff --git a/docs/admin/config/settings.mdx b/docs/admin/config/settings.mdx index 0778de46b..8b81d43ae 100644 --- a/docs/admin/config/settings.mdx +++ b/docs/admin/config/settings.mdx @@ -27,7 +27,7 @@ Settings options and their default values are shown below. {/* SCHEMA_SYNC_START: admin/config/settings.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-06-17T20:29:01Z */} +{/* Last updated: 2026-07-02T18:22:42Z */} ```json { @@ -89,8 +89,8 @@ Settings options and their default values are shown below. // Experimental features and settings. "experimentalFeatures": { + "batchChangeAgentsDisabledDefaultSkills": [], "batchChangeAgentsGuidance": [], - "batchChangesExecution": true, "boostRelevantRepositories": true, "clientSearchResultRanking": "by-zoekt-ranking", "codeInsightsRepoUI": "single-search-query", diff --git a/docs/admin/config/site-config.mdx b/docs/admin/config/site-config.mdx index ee575cacf..6e91ff046 100644 --- a/docs/admin/config/site-config.mdx +++ b/docs/admin/config/site-config.mdx @@ -21,7 +21,7 @@ All site configuration options and their default values are shown below. {/* SCHEMA_SYNC_START: admin/config/site.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-06-17T20:29:01Z */} +{/* Last updated: 2026-07-02T18:22:42Z */} ```json { @@ -45,6 +45,9 @@ All site configuration options and their default values are shown below. // Valid options: "permissive", "enforced" "attribution.mode": "permissive", + // Enables/disables Agentic Batch Changes. Disabled by default. When explicitly set to true, Agentic Batch Changes are enabled only if Batch Changes is licensed and the license includes AI Terms 2025. When unset or false, Agentic Batch Changes are disabled. + "batchChanges.agenticBatchChanges.enabled": false, + // Automatically delete branches created for Batch Changes changesets when the changeset is merged or closed, for supported code hosts. Overrides any setting on the repository on the code host itself. "batchChanges.autoDeleteBranch": false, @@ -88,7 +91,8 @@ All site configuration options and their default values are shown below. // When enabled, only site admins can merge changesets using the Batch Changes UI. This is useful when the Batch Changes GitHub App has elevated access and you want to restrict who can merge changes. "batchChanges.restrictMergeToAdmins": false, - // When enabled, only site admins can create and apply batch changes. + // ⚠️ DEPRECATED: DEPRECATED: This setting has no effect, use RBAC instead + // DEPRECATED: This setting has no effect, use RBAC instead "batchChanges.restrictToAdmins": false, // Specifies specific windows, which can have associated rate limits, to be used when reconciling published changesets (creating or updating). All days and times are handled in UTC. @@ -271,7 +275,7 @@ All site configuration options and their default values are shown below. // Experimental features and settings. "experimentalFeatures": { - "batchChanges.enableAgentWebSearch": false, + "batchChanges.enableAgentWebSearch": true, "batchChanges.enableForkNameSuffix": false, "batchChanges.enablePerforce": false, "codeintelSyntacticIndexing.enabled": false, diff --git a/docs/admin/repo/perforce.mdx b/docs/admin/repo/perforce.mdx index 849da799a..e786e0c34 100644 --- a/docs/admin/repo/perforce.mdx +++ b/docs/admin/repo/perforce.mdx @@ -228,7 +228,7 @@ With this setting, Sourcegraph will ignore any rules with a host other than `*`, {/* SCHEMA_SYNC_START: admin/code_hosts/perforce.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-06-17T20:29:01Z */} +{/* Last updated: 2026-07-02T18:22:42Z */} ```json { // If non-null, enforces Perforce depot permissions. diff --git a/docs/admin/telemetry/private-metadata-allowlist.mdx b/docs/admin/telemetry/private-metadata-allowlist.mdx index 965a8b987..f10600566 100644 --- a/docs/admin/telemetry/private-metadata-allowlist.mdx +++ b/docs/admin/telemetry/private-metadata-allowlist.mdx @@ -36,4 +36,5 @@ To learn more, refer to the [telemetry documentation](https://sourcegraph.com/do | `deepsearch` | `search.categorized` | `topics` — LLM-extracted topic keywords for the question (e.g. authentication, batch-changes); used for product usage reporting. | | `admin.users` | `delete` | `userIDs` — Numeric identifiers of users being deleted; needed for audit and analytics of admin user management actions. | | `externalApi` | `request` | `procedure` — ConnectRPC procedure path (e.g. '/sourcegraph.users.v1.UsersService/GetUser'); not sensitive and needed to distinguish which external API RPCs are being used. | -| `metering` | `consume` | `meteringEventId` — The UUID of the billing event in the metering export queue, used to correlate billing events with telemetry events. | \ No newline at end of file +| `metering` | `consume` | `meteringEventId` — The UUID of the billing event in the metering export queue, used to correlate billing events with telemetry events. | +| `survey.happiness` | `submitted` | `feedback` — Free-text user feedback from the happiness survey; user agrees to submit these to us. | \ No newline at end of file diff --git a/docs/cody/capabilities/supported-models.mdx b/docs/cody/capabilities/supported-models.mdx index 21acfeb7f..5abc3a5fa 100644 --- a/docs/cody/capabilities/supported-models.mdx +++ b/docs/cody/capabilities/supported-models.mdx @@ -17,6 +17,7 @@ Cody supports a variety of cutting-edge large language models for use in chat an | Anthropic | [Claude Opus 4.8](https://platform.claude.com/docs/en/about-claude/models/overview) | ✅ | ✅ | | Anthropic | [Claude Opus 4.7](https://platform.claude.com/docs/en/about-claude/models/overview) | ✅ | ✅ | | Anthropic | [Claude Opus 4.6](https://platform.claude.com/docs/en/about-claude/models/overview) | ✅ | ✅ | +| Anthropic | [Claude Sonnet 5](https://platform.claude.com/docs/en/about-claude/models/whats-new-sonnet-5) | ✅ | ✅ | | Anthropic | [Claude Sonnet 4.6](https://platform.claude.com/docs/en/about-claude/models/overview) | ✅ | ✅ | | Anthropic | [Claude Sonnet 4.6 with Thinking](https://platform.claude.com/docs/en/about-claude/models/overview) | ✅ | ✅ | | Anthropic | [Claude Sonnet 4.5](https://platform.claude.com/docs/en/about-claude/models/overview) | ✅ | ✅ | @@ -27,9 +28,9 @@ Cody supports a variety of cutting-edge large language models for use in chat an | Anthropic | [Claude Haiku 4.5 with Thinking](https://platform.claude.com/docs/en/about-claude/models/overview) | ✅ | ✅ | | Google | [Gemini 2.5 Flash](https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-flash) | ✅ | ✅ | | Google | [Gemini 2.5 Pro](https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-pro) | ✅ | ✅ | -| Google | [Gemini 3 Flash](https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/3-flash) | ✅ | ❌ | -| Google | [Gemini 3.1 Flash Lite](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/3-1-flash-lite) | ✅ | ❌ | +| Google | [Gemini 3.1 Flash Lite](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/gemini/3-1-flash-lite) | ✅ | ✅ | | Google | [Gemini 3.1 Pro](https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/3-1-pro) | ✅ (beta) | ❌ | +| Google | [Gemini 3.5 Flash](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/gemini/3-5-flash) | ✅ | ✅ | | OpenAI | [GPT-5.4](https://developers.openai.com/api/docs/models/gpt-5.4) | ✅ | ✅ | | OpenAI | [GPT-5.4 mini](https://developers.openai.com/api/docs/models/gpt-5.4-mini) | ✅ | ✅ | | OpenAI | [GPT-5.4 nano](https://developers.openai.com/api/docs/models/gpt-5.4-nano) | ✅ | ✅ | diff --git a/docs/self-hosted/observability/alerts.mdx b/docs/self-hosted/observability/alerts.mdx index b34e75d3d..8c7f23bde 100644 --- a/docs/self-hosted/observability/alerts.mdx +++ b/docs/self-hosted/observability/alerts.mdx @@ -1419,37 +1419,6 @@ Generated query for warning alert: `max((max(src_gitserver_echo_duration_seconds
-## gitserver: repo_corrupted - -

number of times a repo corruption has been identified

- -**Descriptions** - -- critical gitserver: 0+ number of times a repo corruption has been identified - -**Next steps** - -- Check the corruption logs for details. gitserver_repos.corruption_logs contains more information. -- More help interpreting this metric is available in the [dashboards reference](dashboards#gitserver-repo_corrupted). -- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: - -```json -"observability.silenceAlerts": [ - "critical_gitserver_repo_corrupted" -] -``` - -*Managed by the Sourcegraph Services team.* - -
-Technical details - -Generated query for critical alert: `max((sum(rate(src_gitserver_repo_corrupted[5m]))) > 0)` - -
- -
- ## gitserver: repository_clone_queue_size

repository clone queue size

@@ -3114,6 +3083,40 @@ Generated query for critical alert: `(min((sum(src_worker_jobs{job=~"^worker.*",
+## worker: codeintel_commit_graph_fast_path_oldest_missing_upload_age_seconds + +

oldest recent completed SCIP upload age missing from the commit graph fast path

+ +**Descriptions** + +- warning worker: 21600s+ oldest recent completed SCIP upload age missing from the commit graph fast path for 15m0s + +**Next steps** + +- Completed, unexpired SCIP uploads finished 1-48h ago are not represented in the persisted commit graph fast path. +- Check whether the codeintel commit graph updater worker is running and healthy. +- Check worker logs for commit graph update failures. +- Inspect codeintel commit graph freshness and upload processing panels for related backlog or errors. +- More help interpreting this metric is available in the [dashboards reference](dashboards#worker-codeintel_commit_graph_fast_path_oldest_missing_upload_age_seconds). +- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: + +```json +"observability.silenceAlerts": [ + "warning_worker_codeintel_commit_graph_fast_path_oldest_missing_upload_age_seconds" +] +``` + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Generated query for warning alert: `max((max(src_codeintel_commit_graph_fast_path_oldest_missing_upload_age_seconds{job=~"^worker.*"\} and on (instance, job) (increase(src_codeintel_commit_graph_fast_path_reporter_total\{job=~"^worker.*"}[15m]) > 0)) or on () vector(0)) > 21600)` + +
+ +
+ ## worker: src_repoupdater_max_sync_backoff

time since oldest sync

@@ -7135,9 +7138,9 @@ Generated query for critical alert: `max((max(deriv(src_deepsearch_questions_in_
Technical details -Generated query for warning alert: `max((sum(rate(src_deepsearch_question_processing_errors_total{operation="question"\}[5m])) / (sum(rate(src_deepsearch_question_processing_total\{operation="question"}[5m])) > 0) * 100) > 10)` +Custom query for warning alert: `max(((sum(increase(src_deepsearch_question_processing_errors_total{operation="question"\}[5m])) / (sum(increase(src_deepsearch_question_processing_total\{operation="question"\}[5m])) + sum(increase(src_deepsearch_question_processing_errors_total\{operation="question"\}[5m]))) * 100) and (sum(increase(src_deepsearch_question_processing_total\{operation="question"\}[5m])) + sum(increase(src_deepsearch_question_processing_errors_total\{operation="question"}[5m])) >= 10)) > 10)` -Generated query for critical alert: `max((sum(rate(src_deepsearch_question_processing_errors_total{operation="question"\}[5m])) / (sum(rate(src_deepsearch_question_processing_total\{operation="question"}[5m])) > 0) * 100) > 20)` +Custom query for critical alert: `max(((sum(increase(src_deepsearch_question_processing_errors_total{operation="question"\}[5m])) / (sum(increase(src_deepsearch_question_processing_total\{operation="question"\}[5m])) + sum(increase(src_deepsearch_question_processing_errors_total\{operation="question"\}[5m]))) * 100) and (sum(increase(src_deepsearch_question_processing_total\{operation="question"\}[5m])) + sum(increase(src_deepsearch_question_processing_errors_total\{operation="question"}[5m])) >= 10)) > 20)`
@@ -7217,6 +7220,147 @@ Generated query for critical alert: `max((sum(increase(src_deepsearch_question_o
+## batch-change-agents: batch_change_agent_message_error_rate + +

message processing error rate over 5m

+ +**Descriptions** + +- warning batch-change-agents: 20%+ message processing error rate over 5m for 15m0s +- critical batch-change-agents: 50%+ message processing error rate over 5m for 15m0s + +**Next steps** + +- Check worker logs for "batch change agent message failed" to find the failing messageIDs and underlying errors. +- Check for upstream LLM provider issues or timeouts (BCA_CONVERSATION_REQUEST_TIMEOUT). +- A spike that correlates with resets in the "Stalled job resetters" group points at worker crashes rather than per-message failures. +- More help interpreting this metric is available in the [dashboards reference](dashboards#batch-change-agents-batch_change_agent_message_error_rate). +- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: + +```json +"observability.silenceAlerts": [ + "warning_batch-change-agents_batch_change_agent_message_error_rate", + "critical_batch-change-agents_batch_change_agent_message_error_rate" +] +``` + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Custom query for warning alert: `max(((sum(increase(src_batch_change_agent_message_errors_total{op="Handle"\}[5m])) / (sum(increase(src_batch_change_agent_message_total\{op="Handle"\}[5m])) + sum(increase(src_batch_change_agent_message_errors_total\{op="Handle"\}[5m]))) * 100) and ((sum(increase(src_batch_change_agent_message_total\{op="Handle"\}[5m])) + sum(increase(src_batch_change_agent_message_errors_total\{op="Handle"}[5m]))) >= 5)) > 20)` + +Custom query for critical alert: `max(((sum(increase(src_batch_change_agent_message_errors_total{op="Handle"\}[5m])) / (sum(increase(src_batch_change_agent_message_total\{op="Handle"\}[5m])) + sum(increase(src_batch_change_agent_message_errors_total\{op="Handle"\}[5m]))) * 100) and ((sum(increase(src_batch_change_agent_message_total\{op="Handle"\}[5m])) + sum(increase(src_batch_change_agent_message_errors_total\{op="Handle"}[5m]))) >= 5)) > 50)` + +
+ +
+ +## batch-change-agents: batch_change_agent_message_queue_wait + +

oldest queued message wait time

+ +**Descriptions** + +- warning batch-change-agents: 300s+ oldest queued message wait time for 15m0s +- critical batch-change-agents: 900s+ oldest queued message wait time for 15m0s + +**Next steps** + +- This is the age of the oldest message still waiting for a free handler, i.e. the latency a user experiences before their agent starts responding. +- Check "batch_change_agent_message_active_handlers": if it is pinned near 100, the pool is saturated and the worker is under-provisioned for current load. +- Check the "Message worker" duration panels for individual messages running unusually long and holding handlers. +- Confirm the worker is healthy and not crash-looping (see the "Stalled job resetters" group). +- More help interpreting this metric is available in the [dashboards reference](dashboards#batch-change-agents-batch_change_agent_message_queue_wait). +- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: + +```json +"observability.silenceAlerts": [ + "warning_batch-change-agents_batch_change_agent_message_queue_wait", + "critical_batch-change-agents_batch_change_agent_message_queue_wait" +] +``` + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Generated query for warning alert: `max((max(src_workerutil_queue_duration_seconds{domain="batch_change_agent_message_worker"})) > 300)` + +Generated query for critical alert: `max((max(src_workerutil_queue_duration_seconds{domain="batch_change_agent_message_worker"})) > 900)` + +
+ +
+ +## batch-change-agents: batch_change_agent_records_reset + +

stalled jobs reset to queued over 1h by queue

+ +**Descriptions** + +- warning batch-change-agents: 5+ stalled jobs reset to queued over 1h by queue for 15m0s + +**Next steps** + +- Resets happen when a worker holding a job stops sending heartbeats, i.e. the worker process crashed or was killed mid-job. +- A small number around a deploy or restart is expected. Sustained resets indicate a crash loop or OOM-kills in the worker. +- Cross-reference worker container restarts and memory usage, and check worker logs for panics. +- More help interpreting this metric is available in the [dashboards reference](dashboards#batch-change-agents-batch_change_agent_records_reset). +- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: + +```json +"observability.silenceAlerts": [ + "warning_batch-change-agents_batch_change_agent_records_reset" +] +``` + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Generated query for warning alert: `max((sum by (domain) (increase(src_dbworker_resetter_record_resets_total{domain=~"^batch_change_agent_.*_resetter$"}[1h]))) > 5)` + +
+ +
+ +## batch-change-agents: batch_change_agent_message_stream_publish_errors + +

message stream publish errors per second

+ +**Descriptions** + +- warning batch-change-agents: 0+ message stream publish errors per second for 5m0s + +**Next steps** + +- These are publish attempts that failed after retries were exhausted, so the corresponding live event was dropped and clients only recover it on reconnect/refetch. +- Check Redis health and connectivity from the worker (the redispool.Store keyspace). +- Confirm the message processing itself is healthy via the "Message worker" group; stream publish failures do not fail the underlying message. +- More help interpreting this metric is available in the [dashboards reference](dashboards#batch-change-agents-batch_change_agent_message_stream_publish_errors). +- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: + +```json +"observability.silenceAlerts": [ + "warning_batch-change-agents_batch_change_agent_message_stream_publish_errors" +] +``` + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Generated query for warning alert: `max((sum(rate(src_batch_change_agents_stream_publish_errors_total[5m]))) > 0)` + +
+ +
+ ## externalapi: externalapi_error_rate

error rate over 5m

diff --git a/docs/self-hosted/observability/dashboards.mdx b/docs/self-hosted/observability/dashboards.mdx index 5b9495677..f7d09d118 100644 --- a/docs/self-hosted/observability/dashboards.mdx +++ b/docs/self-hosted/observability/dashboards.mdx @@ -5935,32 +5935,6 @@ max(src_gitserver_echo_duration_seconds)
-#### gitserver: repo_corrupted - -

Number of times a repo corruption has been identified

- -A non-null value here indicates that a problem has been detected with the gitserver repository storage. -Repository corruptions are never expected. This is a real issue. Gitserver should try to recover from them -by recloning repositories, but this may take a while depending on repo size. - -Refer to the [alerts reference](alerts#gitserver-repo_corrupted) for 1 alert related to this panel. - -To see this panel, visit `/-/debug/grafana/d/gitserver/gitserver?viewPanel=100021` on your Sourcegraph instance. - -*Managed by the Sourcegraph Services team.* - -
-Technical details - -Query: - -``` -sum(rate(src_gitserver_repo_corrupted[5m])) -``` -
- -
- #### gitserver: repository_clone_queue_size

Repository clone queue size

@@ -13928,6 +13902,80 @@ max by (database) (src_dbanalyze_running{job=~"^worker.*"})
+### Worker: Database live/dead tuple monitor + +#### worker: pg_codeintel_table_live_tuples + +

Code intel table live tuples

+ +Estimated number of live rows in each codeintel-db SCIP table, as reported by pg_stat_user_tables. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100200` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +max by (datname, relname)(((src_codeintel_scip_table_tuples{state="live", relname=~"codeintel_scip_.*"}) and on(instance, job) (increase(src_codeintel_scip_table_tuples_reporter_total{job=~"^worker.*"}[15m]) > 0))) +``` +
+ +
+ +#### worker: pg_codeintel_table_dead_tuples + +

Code intel table dead tuples

+ +Estimated number of dead rows in each codeintel-db SCIP table, as reported by pg_stat_user_tables. High values may indicate vacuum is not keeping up with code intel data churn. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100201` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +max by (datname, relname)(((src_codeintel_scip_table_tuples{state="dead", relname=~"codeintel_scip_.*"}) and on(instance, job) (increase(src_codeintel_scip_table_tuples_reporter_total{job=~"^worker.*"}[15m]) > 0))) +``` +
+ +
+ +#### worker: pg_codeintel_table_dead_tuple_percentage + +

Code intel table dead tuples percentage

+ +Estimated dead rows as a percentage of total estimated rows in each codeintel-db SCIP table. Use this with table size and bloat panels to decide whether vacuum/analyze behavior needs investigation. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100202` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +max by (datname, relname)(((src_codeintel_scip_table_tuples{state="dead", relname=~"codeintel_scip_.*"}) and on(instance, job) (increase(src_codeintel_scip_table_tuples_reporter_total{job=~"^worker.*"}[15m]) > 0)) / on(instance, job, datname, relname) clamp_min(((src_codeintel_scip_table_tuples{state="live", relname=~"codeintel_scip_.*"}) and on(instance, job) (increase(src_codeintel_scip_table_tuples_reporter_total{job=~"^worker.*"}[15m]) > 0)) + on(instance, job, datname, relname) ((src_codeintel_scip_table_tuples{state="dead", relname=~"codeintel_scip_.*"}) and on(instance, job) (increase(src_codeintel_scip_table_tuples_reporter_total{job=~"^worker.*"}[15m]) > 0)), 1)) * 100 +``` +
+ +
+ ### Worker: Database record encrypter #### worker: records_encrypted_at_rest_percentage @@ -13938,7 +13986,7 @@ Percentage of encrypted database records This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100200` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100300` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -13962,7 +14010,7 @@ Number of encrypted database records every 5m This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100201` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100301` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -13986,7 +14034,7 @@ Number of encrypted database records every 5m This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100202` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100302` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -14010,7 +14058,7 @@ Number of database record encryption/decryption errors every 5m This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100203` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100303` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -14026,6 +14074,128 @@ sum(increase(src_record_encryption_errors_total{job=~"^worker.*"}[5m]))
+### Worker: Code intelligence commit graph fast path + +#### worker: codeintel_commit_graph_fast_path_affected_repositories + +

Repositories with recent completed SCIP uploads missing from the commit graph fast path

+ +A non-zero value indicates that completed, unexpired SCIP uploads finished 1-48h ago may be stale, missing, or slower to resolve through the primary persisted lookup path. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100400` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +max(src_codeintel_commit_graph_fast_path_affected_repositories{job=~"^worker.*"} and on(instance, job) (increase(src_codeintel_commit_graph_fast_path_reporter_total{job=~"^worker.*"}[15m]) > 0)) or on() vector(0) +``` +
+ +
+ +#### worker: codeintel_commit_graph_fast_path_missing_uploads_total + +

Recent completed SCIP uploads missing from the commit graph fast path

+ +Total number of completed, unexpired SCIP uploads finished 1-48h ago that are missing from the persisted commit graph fast path. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100401` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +max(src_codeintel_commit_graph_fast_path_missing_uploads_total{job=~"^worker.*"} and on(instance, job) (increase(src_codeintel_commit_graph_fast_path_reporter_total{job=~"^worker.*"}[15m]) > 0)) or on() vector(0) +``` +
+ +
+ +#### worker: codeintel_commit_graph_fast_path_max_missing_uploads_per_repository + +

Maximum recent completed SCIP uploads missing from the commit graph fast path for one repository

+ +Largest number of completed, unexpired SCIP uploads finished 1-48h ago that are missing from the persisted commit graph fast path for a single repository. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100402` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +max(src_codeintel_commit_graph_fast_path_max_missing_uploads_per_repository{job=~"^worker.*"} and on(instance, job) (increase(src_codeintel_commit_graph_fast_path_reporter_total{job=~"^worker.*"}[15m]) > 0)) or on() vector(0) +``` +
+ +
+ +#### worker: codeintel_commit_graph_fast_path_oldest_missing_upload_age_seconds + +

Oldest recent completed SCIP upload age missing from the commit graph fast path

+ +Age in seconds of the oldest completed, unexpired SCIP upload finished 1-48h ago that is missing from the persisted commit graph fast path. + +Refer to the [alerts reference](alerts#worker-codeintel_commit_graph_fast_path_oldest_missing_upload_age_seconds) for 1 alert related to this panel. + +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100410` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +max(src_codeintel_commit_graph_fast_path_oldest_missing_upload_age_seconds{job=~"^worker.*"} and on(instance, job) (increase(src_codeintel_commit_graph_fast_path_reporter_total{job=~"^worker.*"}[15m]) > 0)) or on() vector(0) +``` +
+ +
+ +#### worker: codeintel_commit_graph_fast_path_newest_missing_upload_age_seconds + +

Newest recent completed SCIP upload age missing from the commit graph fast path

+ +Age in seconds of the newest completed, unexpired SCIP upload finished 1-48h ago that is missing from the persisted commit graph fast path. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100411` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +max(src_codeintel_commit_graph_fast_path_newest_missing_upload_age_seconds{job=~"^worker.*"} and on(instance, job) (increase(src_codeintel_commit_graph_fast_path_reporter_total{job=~"^worker.*"}[15m]) > 0)) or on() vector(0) +``` +
+ +
+ ### Worker: Codeintel: Repository commit graph updates #### worker: codeintel_commit_graph_processor_total @@ -14034,7 +14204,7 @@ sum(increase(src_record_encryption_errors_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100300` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100500` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14056,7 +14226,7 @@ sum(increase(src_codeintel_commit_graph_processor_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100301` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100501` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14078,7 +14248,7 @@ sum by (le)(rate(src_codeintel_commit_graph_processor_duration_seconds_bucket{j This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100302` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100502` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14100,7 +14270,7 @@ sum(increase(src_codeintel_commit_graph_processor_errors_total{job=~"^worker.*"} This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100303` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100503` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14124,7 +14294,7 @@ sum(increase(src_codeintel_commit_graph_processor_errors_total{job=~"^worker.*"} This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100400` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100600` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14146,7 +14316,7 @@ sum(increase(src_codeintel_autoindexing_total{op='HandleIndexSchedule',job=~"^wo This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100401` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100601` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14168,7 +14338,7 @@ sum by (le)(rate(src_codeintel_autoindexing_duration_seconds_bucket{op='HandleI This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100402` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100602` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14190,7 +14360,7 @@ sum(increase(src_codeintel_autoindexing_errors_total{op='HandleIndexSchedule',jo This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100403` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100603` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14214,7 +14384,7 @@ sum(increase(src_codeintel_autoindexing_errors_total{op='HandleIndexSchedule',jo This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100500` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100700` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14236,7 +14406,7 @@ sum(increase(src_codeintel_uploads_store_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100501` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100701` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14258,7 +14428,7 @@ sum by (le)(rate(src_codeintel_uploads_store_duration_seconds_bucket{job=~"^wor This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100502` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100702` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14280,7 +14450,7 @@ sum(increase(src_codeintel_uploads_store_errors_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100503` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100703` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14302,7 +14472,7 @@ sum(increase(src_codeintel_uploads_store_errors_total{job=~"^worker.*"}[5m])) / This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100510` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100710` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14324,7 +14494,7 @@ sum by (op)(increase(src_codeintel_uploads_store_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100511` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100711` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14346,7 +14516,7 @@ histogram_quantile(0.99, sum by (le,op)(rate(src_codeintel_uploads_store_durati This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100512` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100712` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14368,7 +14538,7 @@ sum by (op)(increase(src_codeintel_uploads_store_errors_total{job=~"^worker.*"}[ This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100513` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100713` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14392,7 +14562,7 @@ sum by (op)(increase(src_codeintel_uploads_store_errors_total{job=~"^worker.*"}[ This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100600` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100800` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14414,7 +14584,7 @@ sum(increase(src_codeintel_uploads_lsifstore_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100601` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100801` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14436,7 +14606,7 @@ sum by (le)(rate(src_codeintel_uploads_lsifstore_duration_seconds_bucket{job=~" This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100602` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100802` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14458,7 +14628,7 @@ sum(increase(src_codeintel_uploads_lsifstore_errors_total{job=~"^worker.*"}[5m]) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100603` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100803` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14480,7 +14650,7 @@ sum(increase(src_codeintel_uploads_lsifstore_errors_total{job=~"^worker.*"}[5m]) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100610` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100810` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14502,7 +14672,7 @@ sum by (op)(increase(src_codeintel_uploads_lsifstore_total{job=~"^worker.*"}[5m] This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100611` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100811` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14524,7 +14694,7 @@ histogram_quantile(0.99, sum by (le,op)(rate(src_codeintel_uploads_lsifstore_du This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100612` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100812` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14546,7 +14716,7 @@ sum by (op)(increase(src_codeintel_uploads_lsifstore_errors_total{job=~"^worker. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100613` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100813` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14570,7 +14740,7 @@ sum by (op)(increase(src_codeintel_uploads_lsifstore_errors_total{job=~"^worker. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100700` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100900` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14592,7 +14762,7 @@ sum(increase(src_gitserver_client_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100701` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100901` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14614,7 +14784,7 @@ sum by (le)(rate(src_gitserver_client_duration_seconds_bucket{job=~"^worker.*"} This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100702` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100902` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14636,7 +14806,7 @@ sum(increase(src_gitserver_client_errors_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100703` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100903` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14658,7 +14828,7 @@ sum(increase(src_gitserver_client_errors_total{job=~"^worker.*"}[5m])) / (sum(in This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100710` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100910` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14680,7 +14850,7 @@ sum by (op)(increase(src_gitserver_client_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100711` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100911` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14702,7 +14872,7 @@ histogram_quantile(0.99, sum by (le,op)(rate(src_gitserver_client_duration_seco This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100712` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100912` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14724,7 +14894,7 @@ sum by (op)(increase(src_gitserver_client_errors_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100713` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100913` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -14751,7 +14921,7 @@ If the value is persistently high, make sure all external services have valid to This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100800` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101000` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -14773,7 +14943,7 @@ max(timestamp(vector(time()))) - max(src_repoupdater_syncer_sync_last_time) Refer to the [alerts reference](alerts#worker-src_repoupdater_max_sync_backoff) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100801` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101001` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -14795,7 +14965,7 @@ max(src_repoupdater_max_sync_backoff) Refer to the [alerts reference](alerts#worker-src_repoupdater_syncer_sync_errors_total) for 2 alerts related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100802` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101002` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -14817,7 +14987,7 @@ max by (family) (rate(src_repoupdater_syncer_sync_errors_total{owner!="user",rea Refer to the [alerts reference](alerts#worker-syncer_sync_start) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100810` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101010` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -14839,7 +15009,7 @@ max by (family) (rate(src_repoupdater_syncer_start_sync{family="Syncer.SyncExter Refer to the [alerts reference](alerts#worker-syncer_sync_duration) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100811` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101011` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -14861,7 +15031,7 @@ histogram_quantile(0.95, max by (le, family, success) (rate(src_repoupdater_sync Refer to the [alerts reference](alerts#worker-source_duration) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100812` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101012` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -14883,7 +15053,7 @@ histogram_quantile(0.95, max by (le) (rate(src_repoupdater_source_duration_secon Refer to the [alerts reference](alerts#worker-syncer_synced_repos) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100820` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101020` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -14905,7 +15075,7 @@ max(rate(src_repoupdater_syncer_synced_repos_total[1m])) Refer to the [alerts reference](alerts#worker-sourced_repos) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100821` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101021` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -14927,7 +15097,7 @@ max(rate(src_repoupdater_source_repos_total[1m])) Refer to the [alerts reference](alerts#worker-sched_auto_fetch) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100830` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101030` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -14952,7 +15122,7 @@ This does not indicate anything if there are no user added code hosts. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100831` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101031` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -14974,7 +15144,7 @@ max(rate(src_repoupdater_sched_manual_fetch[1m])) Refer to the [alerts reference](alerts#worker-sched_loops) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100840` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101040` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -14996,7 +15166,7 @@ max(rate(src_repoupdater_sched_loops[1m])) Refer to the [alerts reference](alerts#worker-src_repoupdater_stale_repos) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100841` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101041` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15018,7 +15188,7 @@ max(src_repoupdater_stale_repos) Refer to the [alerts reference](alerts#worker-sched_error) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100842` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101042` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15040,7 +15210,7 @@ max(rate(src_repoupdater_sched_error[1m])) Refer to the [alerts reference](alerts#worker-src_repoupdater_cleanup_failed_repos) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100843` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101043` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15066,7 +15236,7 @@ max(src_repoupdater_cleanup_failed_repos) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100900` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101100` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15094,7 +15264,7 @@ were correct. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100901` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101101` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15119,7 +15289,7 @@ Check the reasons for why they were deleted. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100902` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101102` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15144,7 +15314,7 @@ Check the reasons for why they were deleted. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=100903` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101103` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15168,7 +15338,7 @@ sum by (reason) (src_repo_statesyncer_repos_deleted{is_primary="false"}) Refer to the [alerts reference](alerts#worker-src_repoupdater_external_services_total) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101000` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101200` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15190,7 +15360,7 @@ max(src_repoupdater_external_services_total) Refer to the [alerts reference](alerts#worker-repoupdater_queued_sync_jobs_total) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101010` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101210` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15212,7 +15382,7 @@ max(src_repoupdater_queued_sync_jobs_total) Refer to the [alerts reference](alerts#worker-repoupdater_completed_sync_jobs_total) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101011` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101211` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15234,7 +15404,7 @@ max(src_repoupdater_completed_sync_jobs_total) Refer to the [alerts reference](alerts#worker-repoupdater_errored_sync_jobs_percentage) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101012` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101212` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15256,7 +15426,7 @@ max(src_repoupdater_errored_sync_jobs_percentage) Refer to the [alerts reference](alerts#worker-github_graphql_rate_limit_remaining) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101020` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101220` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15278,7 +15448,7 @@ max by (name) (src_github_rate_limit_remaining_v2{resource="graphql"}) Refer to the [alerts reference](alerts#worker-github_rest_rate_limit_remaining) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101021` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101221` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15300,7 +15470,7 @@ max by (name) (src_github_rate_limit_remaining_v2{resource="rest"}) Refer to the [alerts reference](alerts#worker-github_search_rate_limit_remaining) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101022` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101222` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15324,7 +15494,7 @@ Indicates how long we`re waiting on the rate limit once it has been exceeded This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101030` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101230` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15348,7 +15518,7 @@ Indicates how long we`re waiting on the rate limit once it has been exceeded This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101031` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101231` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15372,7 +15542,7 @@ Indicates how long we`re waiting on the rate limit once it has been exceeded This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101032` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101232` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15394,7 +15564,7 @@ max by(name) (rate(src_github_rate_limit_wait_duration_seconds{resource="search" Refer to the [alerts reference](alerts#worker-gitlab_rest_rate_limit_remaining) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101040` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101240` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15418,7 +15588,7 @@ Indicates how long we`re waiting on the rate limit once it has been exceeded This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101041` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101241` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15442,7 +15612,7 @@ Indicates how long we`re waiting on our internal rate limiter when communicating This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101050` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101250` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15466,7 +15636,7 @@ The rate at which we fail our internal rate limiter. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101051` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101251` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15492,7 +15662,7 @@ Indicates the total number of user permissions sync completed. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101100` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101300` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15516,7 +15686,7 @@ Indicates the number of users permissions syncs completed. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101101` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101301` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15540,7 +15710,7 @@ Indicates the number of permissions syncs done for the first time for the user. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101102` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101302` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15564,7 +15734,7 @@ Indicates the total number of repo permissions sync completed. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101110` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101310` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15588,7 +15758,7 @@ Indicates the number of repos permissions syncs completed. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101111` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101311` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15612,7 +15782,7 @@ Indicates the number of permissions syncs done for the first time for the repo. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101112` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101312` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15636,7 +15806,7 @@ Indicates the max delay between two consecutive permissions sync for a user duri This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101120` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101320` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15660,7 +15830,7 @@ Indicates the max delay between two consecutive permissions sync for a repo duri This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101121` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101321` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15684,7 +15854,7 @@ Indicates the max delay between user creation and their permissions sync This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101130` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101330` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15708,7 +15878,7 @@ Indicates the max delay between repo creation and their permissions sync This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101131` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101331` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15732,7 +15902,7 @@ Indicates the number permissions found during users/repos permissions sync. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101140` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101340` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15756,7 +15926,7 @@ Indicates the average number permissions found during permissions sync per user/ This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101141` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101341` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15778,7 +15948,7 @@ avg by (type) (src_repo_perms_syncer_perms_found) Refer to the [alerts reference](alerts#worker-perms_syncer_outdated_perms) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101150` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101350` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15800,7 +15970,7 @@ max by (type) (src_repo_perms_syncer_outdated_perms) Refer to the [alerts reference](alerts#worker-perms_syncer_sync_duration) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101160` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101360` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15826,7 +15996,7 @@ Permissions sync errors are often transient and rarely actionable. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101170` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101370` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15844,14 +16014,14 @@ max by (type) (ceil(rate(src_repo_perms_syncer_sync_errors_total[1m]))) #### worker: perms_syncer_scheduled_repos_total -

Total number of repos scheduled for permissions sync

+

Rate of users/repos scheduled for permissions sync

-Indicates how many repositories have been scheduled for a permissions sync. +Indicates the per-minute rate at which users/repositories are scheduled for a permissions sync. More about repository permissions synchronization [here](https://sourcegraph.com/docs/admin/permissions/syncing#scheduling) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101171` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101371` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15861,7 +16031,7 @@ To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101171` on Query: ``` -max(rate(src_repo_perms_syncer_schedule_repos_total[1m])) +sum by (type) (increase(src_repo_perms_syncer_schedule_total[1m])) ``` @@ -15875,7 +16045,7 @@ max(rate(src_repo_perms_syncer_schedule_repos_total[1m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101200` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101400` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15897,7 +16067,7 @@ sum(increase(src_gitserver_client_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101201` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101401` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15919,7 +16089,7 @@ sum by (le)(rate(src_gitserver_client_duration_seconds_bucket{job=~"^worker.*"} This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101202` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101402` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15941,7 +16111,7 @@ sum(increase(src_gitserver_client_errors_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101203` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101403` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15963,7 +16133,7 @@ sum(increase(src_gitserver_client_errors_total{job=~"^worker.*"}[5m])) / (sum(in This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101210` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101410` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15985,7 +16155,7 @@ sum by (op,scope)(increase(src_gitserver_client_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101211` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101411` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16007,7 +16177,7 @@ histogram_quantile(0.99, sum by (le,op,scope)(rate(src_gitserver_client_duratio This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101212` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101412` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16029,7 +16199,7 @@ sum by (op,scope)(increase(src_gitserver_client_errors_total{job=~"^worker.*"}[5 This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101213` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101413` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16053,7 +16223,7 @@ sum by (op,scope)(increase(src_gitserver_client_errors_total{job=~"^worker.*"}[5 This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101300` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101500` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16075,7 +16245,7 @@ sum(increase(src_gitserver_repositoryservice_client_total{job=~"^worker.*"}[5m]) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101301` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101501` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16097,7 +16267,7 @@ sum by (le)(rate(src_gitserver_repositoryservice_client_duration_seconds_bucket This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101302` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101502` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16119,7 +16289,7 @@ sum(increase(src_gitserver_repositoryservice_client_errors_total{job=~"^worker.* This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101303` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101503` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16141,7 +16311,7 @@ sum(increase(src_gitserver_repositoryservice_client_errors_total{job=~"^worker.* This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101310` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101510` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16163,7 +16333,7 @@ sum by (op,scope)(increase(src_gitserver_repositoryservice_client_total{job=~"^w This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101311` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101511` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16185,7 +16355,7 @@ histogram_quantile(0.99, sum by (le,op,scope)(rate(src_gitserver_repositoryserv This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101312` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101512` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16207,7 +16377,7 @@ sum by (op,scope)(increase(src_gitserver_repositoryservice_client_errors_total{j This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101313` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101513` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16231,7 +16401,7 @@ sum by (op,scope)(increase(src_gitserver_repositoryservice_client_errors_total{j This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101400` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101600` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16253,7 +16423,7 @@ sum(increase(src_batches_dbstore_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101401` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101601` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16275,7 +16445,7 @@ sum by (le)(rate(src_batches_dbstore_duration_seconds_bucket{job=~"^worker.*"}[ This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101402` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101602` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16297,7 +16467,7 @@ sum(increase(src_batches_dbstore_errors_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101403` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101603` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16319,7 +16489,7 @@ sum(increase(src_batches_dbstore_errors_total{job=~"^worker.*"}[5m])) / (sum(inc This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101410` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101610` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16341,7 +16511,7 @@ sum by (op)(increase(src_batches_dbstore_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101411` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101611` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16363,7 +16533,7 @@ histogram_quantile(0.99, sum by (le,op)(rate(src_batches_dbstore_duration_secon This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101412` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101612` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16385,7 +16555,7 @@ sum by (op)(increase(src_batches_dbstore_errors_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101413` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101613` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16409,7 +16579,7 @@ sum by (op)(increase(src_batches_dbstore_errors_total{job=~"^worker.*"}[5m])) / This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101500` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101700` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16431,7 +16601,7 @@ sum(increase(src_batches_service_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101501` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101701` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16453,7 +16623,7 @@ sum by (le)(rate(src_batches_service_duration_seconds_bucket{job=~"^worker.*"}[ This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101502` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101702` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16475,7 +16645,7 @@ sum(increase(src_batches_service_errors_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101503` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101703` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16497,7 +16667,7 @@ sum(increase(src_batches_service_errors_total{job=~"^worker.*"}[5m])) / (sum(inc This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101510` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101710` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16519,7 +16689,7 @@ sum by (op)(increase(src_batches_service_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101511` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101711` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16541,7 +16711,7 @@ histogram_quantile(0.99, sum by (le,op)(rate(src_batches_service_duration_secon This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101512` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101712` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16563,7 +16733,7 @@ sum by (op)(increase(src_batches_service_errors_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101513` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101713` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16587,7 +16757,7 @@ sum by (op)(increase(src_batches_service_errors_total{job=~"^worker.*"}[5m])) / This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101600` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101800` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16609,7 +16779,7 @@ sum(src_query_runner_worker_processor_handlers{job=~"^worker.*"}) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101610` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101810` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16631,7 +16801,7 @@ sum(increase(src_query_runner_worker_processor_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101611` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101811` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16653,7 +16823,7 @@ sum by (le)(rate(src_query_runner_worker_processor_duration_seconds_bucket{job= This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101612` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101812` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16675,7 +16845,7 @@ sum(increase(src_query_runner_worker_processor_errors_total{job=~"^worker.*"}[5m This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101613` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101813` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16699,7 +16869,7 @@ sum(increase(src_query_runner_worker_processor_errors_total{job=~"^worker.*"}[5m This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101700` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101900` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16721,7 +16891,7 @@ sum(increase(src_workerutil_dbworker_store_total{domain='insights_query_runner_j This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101701` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101901` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16743,7 +16913,7 @@ sum by (le)(rate(src_workerutil_dbworker_store_duration_seconds_bucket{domain=' This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101702` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101902` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16765,7 +16935,7 @@ sum(increase(src_workerutil_dbworker_store_errors_total{domain='insights_query_r This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101703` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101903` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16787,7 +16957,7 @@ sum(increase(src_workerutil_dbworker_store_errors_total{domain='insights_query_r This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101710` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101910` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16809,7 +16979,7 @@ sum by (op)(increase(src_workerutil_dbworker_store_total{domain='insights_query_ This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101711` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101911` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16831,7 +17001,7 @@ histogram_quantile(0.99, sum by (le,op)(rate(src_workerutil_dbworker_store_dura This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101712` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101912` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16853,7 +17023,7 @@ sum by (op)(increase(src_workerutil_dbworker_store_errors_total{domain='insights This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101713` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101913` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16880,7 +17050,7 @@ A value of 0 indicates the routine isn`t running currently, it awaits it`s next This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101800` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102000` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -16905,7 +17075,7 @@ A low or zero value could indicate that a routine is stalled or encountering err This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101801` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102001` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -16930,7 +17100,7 @@ A sustained high error rate may indicate a problem with the routine`s configurat Refer to the [alerts reference](alerts#worker-goroutine_error_rate) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101810` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102010` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16955,7 +17125,7 @@ A value above 5% indicates that a significant portion of routine executions are Refer to the [alerts reference](alerts#worker-goroutine_error_percentage) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101811` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102011` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16980,7 +17150,7 @@ Longer durations might indicate increased load or processing time. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101820` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102020` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17005,7 +17175,7 @@ This represents how long a complete loop iteration takes before sleeping for the This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101821` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102021` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17030,7 +17200,7 @@ Higher values indicate that tenant processing is taking longer and may affect ov This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101830` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102030` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17055,7 +17225,7 @@ Consistently high values might indicate problematic tenants or inefficient proce This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101831` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102031` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17080,7 +17250,7 @@ Unexpected changes can indicate tenant configuration issues or scaling events. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101840` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102040` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17105,7 +17275,7 @@ A healthy routine should maintain a consistent processing rate. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101841` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102041` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17130,7 +17300,7 @@ Consistent errors indicate problems with specific tenants. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101850` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102050` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17155,7 +17325,7 @@ Values above 5% indicate significant tenant processing problems. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101851` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102051` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17179,7 +17349,7 @@ Query: This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101900` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102100` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17201,7 +17371,7 @@ sum by (app_name, db_name) (src_pgsql_conns_max_open{app_name="worker"}) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101901` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102101` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17223,7 +17393,7 @@ sum by (app_name, db_name) (src_pgsql_conns_open{app_name="worker"}) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101910` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102110` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17245,7 +17415,7 @@ sum by (app_name, db_name) (src_pgsql_conns_in_use{app_name="worker"}) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101911` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102111` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17267,7 +17437,7 @@ sum by (app_name, db_name) (src_pgsql_conns_idle{app_name="worker"}) Refer to the [alerts reference](alerts#worker-mean_blocked_seconds_per_conn_request) for 2 alerts related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101920` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102120` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17289,7 +17459,7 @@ sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="w This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101930` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102130` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17311,7 +17481,7 @@ sum by (app_name, db_name) (increase(src_pgsql_conns_closed_max_idle{app_name="w This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101931` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102131` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17333,7 +17503,7 @@ sum by (app_name, db_name) (increase(src_pgsql_conns_closed_max_lifetime{app_nam This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101932` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102132` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17357,7 +17527,7 @@ sum by (app_name, db_name) (increase(src_pgsql_conns_closed_max_idle_time{app_na Refer to the [alerts reference](alerts#worker-cpu_usage_percentage) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102000` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102200` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17381,7 +17551,7 @@ An estimate for the active memory in use, which includes anonymous memory, file This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102001` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102201` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17405,7 +17575,7 @@ An estimate for the active memory in use in bytes, which includes anonymous memo This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102002` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102202` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17429,7 +17599,7 @@ The total anonymous memory in use by the application, which includes Go stack an Refer to the [alerts reference](alerts#worker-memory_rss) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102010` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102210` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17453,7 +17623,7 @@ This metric shows the total active file-backed memory currently in use by the ap This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102011` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102211` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17477,7 +17647,7 @@ The kernel usage metric shows the amount of memory used by the kernel on behalf This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102012` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102212` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17511,7 +17681,7 @@ value change independent of deployment events (such as an upgrade), it could ind This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102100` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102300` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17533,7 +17703,7 @@ count by(name) ((time() - container_last_seen{name=~"^worker.*"}) > 60) Refer to the [alerts reference](alerts#worker-container_cpu_usage) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102101` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102301` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17555,7 +17725,7 @@ cadvisor_container_cpu_usage_percentage_total{name=~"^worker.*"} Refer to the [alerts reference](alerts#worker-container_memory_usage) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102102` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102302` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17580,7 +17750,7 @@ When extremely high, this can indicate a resource usage problem, or can cause pr This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102103` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102303` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17604,7 +17774,7 @@ sum by(name) (rate(container_fs_reads_total{name=~"^worker.*"}[1h]) + rate(conta Refer to the [alerts reference](alerts#worker-provisioning_container_cpu_usage_long_term) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102200` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102400` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -17626,7 +17796,7 @@ quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^wo Refer to the [alerts reference](alerts#worker-provisioning_container_memory_usage_long_term) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102201` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102401` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -17648,7 +17818,7 @@ max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^worker.*" Refer to the [alerts reference](alerts#worker-provisioning_container_cpu_usage_short_term) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102210` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102410` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -17670,7 +17840,7 @@ max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^worker.*"}[5 Refer to the [alerts reference](alerts#worker-provisioning_container_memory_usage_short_term) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102211` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102411` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -17695,7 +17865,7 @@ When it occurs frequently, it is an indicator of underprovisioning. Refer to the [alerts reference](alerts#worker-container_oomkill_events_total) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102212` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102412` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -17721,7 +17891,7 @@ A high value here indicates a possible goroutine leak. Refer to the [alerts reference](alerts#worker-go_goroutines) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102300` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102500` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -17743,7 +17913,7 @@ max by(instance) (go_goroutines{job=~".*worker"}) Refer to the [alerts reference](alerts#worker-go_gc_duration_seconds) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102301` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102501` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -17767,7 +17937,7 @@ max by(instance) (go_gc_duration_seconds{job=~".*worker"}) Refer to the [alerts reference](alerts#worker-pods_available_percentage) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102400` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102600` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -17791,7 +17961,7 @@ sum by(app) (up{app=~".*worker"}) / count by (app) (up{app=~".*worker"}) * 100 This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102500` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102700` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17813,7 +17983,7 @@ sum(increase(src_workerutil_dbworker_store_total{domain='own_background_worker_s This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102501` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102701` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17835,7 +18005,7 @@ sum by (le)(rate(src_workerutil_dbworker_store_duration_seconds_bucket{domain=' This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102502` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102702` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17857,7 +18027,7 @@ sum(increase(src_workerutil_dbworker_store_errors_total{domain='own_background_w This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102503` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102703` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17879,7 +18049,7 @@ sum(increase(src_workerutil_dbworker_store_errors_total{domain='own_background_w This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102510` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102710` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17901,7 +18071,7 @@ sum by (op)(increase(src_workerutil_dbworker_store_total{domain='own_background_ This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102511` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102711` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17923,7 +18093,7 @@ histogram_quantile(0.99, sum by (le,op)(rate(src_workerutil_dbworker_store_dura This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102512` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102712` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17945,7 +18115,7 @@ sum by (op)(increase(src_workerutil_dbworker_store_errors_total{domain='own_back This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102513` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102713` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17969,7 +18139,7 @@ sum by (op)(increase(src_workerutil_dbworker_store_errors_total{domain='own_back This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102600` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102800` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -17991,7 +18161,7 @@ sum(src_own_background_worker_processor_handlers{job=~"^worker.*"}) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102610` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102810` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -18013,7 +18183,7 @@ sum(increase(src_own_background_worker_processor_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102611` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102811` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -18035,7 +18205,7 @@ sum by (le)(rate(src_own_background_worker_processor_duration_seconds_bucket{jo This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102612` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102812` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -18057,7 +18227,7 @@ sum(increase(src_own_background_worker_processor_errors_total{job=~"^worker.*"}[ This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102613` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102813` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -18081,7 +18251,7 @@ sum(increase(src_own_background_worker_processor_errors_total{job=~"^worker.*"}[ This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102700` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102900` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -18103,7 +18273,7 @@ sum by (op)(increase(src_own_background_index_scheduler_total{job=~"^worker.*"}[ This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102701` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102901` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -18125,7 +18295,7 @@ histogram_quantile(0.99, sum by (le,op)(rate(src_own_background_index_scheduler This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102702` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102902` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -18147,7 +18317,7 @@ sum by (op)(increase(src_own_background_index_scheduler_errors_total{job=~"^work This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102703` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102903` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -18173,7 +18343,7 @@ The duration since the configuration client used by the "worker" service last su This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102800` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=103000` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -18195,7 +18365,7 @@ src_conf_client_time_since_last_successful_update_seconds{job=~`^worker.*`,insta Refer to the [alerts reference](alerts#worker-worker_site_configuration_duration_since_last_successful_update_by_instance) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102801` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=103001` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -34879,7 +35049,7 @@ To see this panel, visit `/-/debug/grafana/d/deepsearch/deepsearch?viewPanel=100 Query: ``` -sum(rate(src_deepsearch_question_processing_errors_total{operation="question"}[5m])) / (sum(rate(src_deepsearch_question_processing_total{operation="question"}[5m])) > 0) * 100 +sum(increase(src_deepsearch_question_processing_errors_total{operation="question"}[5m])) / (sum(increase(src_deepsearch_question_processing_total{operation="question"}[5m])) + sum(increase(src_deepsearch_question_processing_errors_total{operation="question"}[5m]))) * 100 ``` @@ -35129,6 +35299,698 @@ sum(increase(src_deepsearch_question_outcome_total{outcome="token_limit_exceeded
+## Batch Change Agents + +

Background processing for batch change agents: the message and wake workers, their queues and resetters, and the event streams that drive live UI updates.

+ +To see this dashboard, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents` on your Sourcegraph instance. + +### Batch Change Agents: Message worker + +#### batch-change-agents: batch_change_agent_message_active_handlers + +

Active message handlers

+ +Number of batch change agent messages being processed concurrently. Each worker is configured for up to 100 concurrent handlers; sustained values near that ceiling mean new messages are waiting for a free slot. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100000` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +sum(src_batch_change_agent_message_handlers) +``` +
+ +
+ +#### batch-change-agents: batch_change_agent_message_processing_rate + +

Messages processed per minute

+ +Rate of successfully processed batch change agent messages. A message is one turn of the agent loop for a user; this counts only successful completions (failures appear in the error rate panel). + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100001` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +sum(rate(src_batch_change_agent_message_total{op="Handle"}[5m])) * 60 +``` +
+ +
+ +#### batch-change-agents: batch_change_agent_message_p95_duration + +

95th percentile message processing duration

+ +95th percentile end-to-end time to process a message (the full agent loop, including all LLM requests). The histogram buckets span 1s to 1h; values pinned at the top bucket mean messages are running longer than an hour. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100010` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +histogram_quantile(0.95, sum(rate(src_batch_change_agent_message_duration_seconds_bucket{op="Handle"}[15m])) by (le)) +``` +
+ +
+ +#### batch-change-agents: batch_change_agent_message_p50_duration + +

50th percentile message processing duration

+ +Median end-to-end time to process a message. Useful for spotting a shift in typical agent-loop latency that the error rate would not reveal. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100011` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +histogram_quantile(0.50, sum(rate(src_batch_change_agent_message_duration_seconds_bucket{op="Handle"}[15m])) by (le)) +``` +
+ +
+ +#### batch-change-agents: batch_change_agent_message_error_rate + +

Message processing error rate over 5m

+ +Percentage of message handler invocations that ended in an error. The alert is gated on at least 5 attempts in the window so it stays quiet during idle periods. + +Refer to the [alerts reference](alerts#batch-change-agents-batch_change_agent_message_error_rate) for 2 alerts related to this panel. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100020` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +sum(increase(src_batch_change_agent_message_errors_total{op="Handle"}[5m])) / (sum(increase(src_batch_change_agent_message_total{op="Handle"}[5m])) + sum(increase(src_batch_change_agent_message_errors_total{op="Handle"}[5m]))) * 100 +``` +
+ +
+ +### Batch Change Agents: Queue backlog + +#### batch-change-agents: batch_change_agent_message_queue_depth + +

Messages waiting in the queue

+ +Number of batch change agent messages enqueued but not yet picked up by a handler. Sustained growth means messages are arriving faster than the 100-handler pool can process them. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100100` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +max(src_workerutil_queue_depth{domain="batch_change_agent_message_worker"}) +``` +
+ +
+ +#### batch-change-agents: batch_change_agent_message_queue_wait + +

Oldest queued message wait time

+ +How long the oldest queued message has been waiting for a handler. Zero when the queue is empty. + +Refer to the [alerts reference](alerts#batch-change-agents-batch_change_agent_message_queue_wait) for 2 alerts related to this panel. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100101` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +max(src_workerutil_queue_duration_seconds{domain="batch_change_agent_message_worker"}) +``` +
+ +
+ +#### batch-change-agents: batch_change_agent_wake_queue_depth + +

Wake jobs waiting in the queue

+ +Number of wake jobs enqueued but not yet processed. These enqueue a message for the message worker, so a backlog here delays idle threads from re-activating. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100110` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +max(src_workerutil_queue_depth{domain="batch_change_agent_wake_worker"}) +``` +
+ +
+ +#### batch-change-agents: batch_change_agent_wake_queue_wait + +

Oldest queued wake job wait time

+ +How long the oldest queued wake job has been waiting for a handler. Zero when the queue is empty. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100111` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +max(src_workerutil_queue_duration_seconds{domain="batch_change_agent_wake_worker"}) +``` +
+ +
+ +### Batch Change Agents: Tool calls + +#### batch-change-agents: batch_change_agent_tool_call_rate + +

Tool calls per minute by tool

+ +Rate of successful tool invocations within the agent loop, broken down by tool name (e.g. prepare_batch_spec, run_workspaces, deepsearch). A tool is counted here once it completes without error; failures appear in the error rate panel. Tools called inside subagents (e.g. the read-only workspace tools the analyze_workspaces subagent runs) are counted here too. This is the per-tool view of what the agent loop spends its turns doing. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100200` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +sum by (tool) (rate(src_batch_change_agent_tool_total[5m])) * 60 +``` +
+ +
+ +#### batch-change-agents: batch_change_agent_tool_error_rate + +

Tool error rate over 5m by tool

+ +Percentage of invocations per tool that returned an error to the agent loop. A consistently hot tool usually points at a flaky downstream integration it wraps (gitserver, the batch-spec executor, deep search) rather than the agent itself; the model often recovers by retrying, so this complements rather than duplicates the message worker error rate. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100201` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +sum by (tool) (increase(src_batch_change_agent_tool_errors_total[5m])) / (sum by (tool) (increase(src_batch_change_agent_tool_total[5m])) + sum by (tool) (increase(src_batch_change_agent_tool_errors_total[5m]))) * 100 +``` +
+ +
+ +#### batch-change-agents: batch_change_agent_tool_p95_duration + +

95th percentile tool call duration by tool

+ +95th percentile execution time per tool. This is the per-tool breakdown of where time in the agent loop goes; long-running tools (deepsearch, wait_for_changesets, wait_for_workspaces) are expected to dominate, while batch-spec and filesystem tools should stay fast. A subagent tool`s duration (e.g. analyze_workspaces) overlaps the inner tool calls it makes, which are also recorded here, so durations should not be summed into a single total. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100202` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +histogram_quantile(0.95, sum by (le, tool) (rate(src_batch_change_agent_tool_duration_seconds_bucket[5m]))) +``` +
+ +
+ +### Batch Change Agents: Wake worker + +#### batch-change-agents: batch_change_agent_wake_active_handlers + +

Active wake handlers

+ +Number of wake jobs being processed concurrently. The wake worker only resolves an agent and enqueues a wake message (the LLM work happens in the message worker), so it runs a small pool of 3 handlers. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100300` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +sum(src_batch_change_agent_wake_handlers) +``` +
+ +
+ +#### batch-change-agents: batch_change_agent_wake_processing_rate + +

Wake jobs processed per minute

+ +Rate of successfully processed wake jobs that re-activate idle threads with unread inbox items. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100301` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +sum(rate(src_batch_change_agent_wake_total{op="Handle"}[5m])) * 60 +``` +
+ +
+ +#### batch-change-agents: batch_change_agent_wake_p95_duration + +

95th percentile wake processing duration

+ +95th percentile time to process a wake job. Wake jobs are a few quick DB operations, so this should stay well under a second. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100302` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +histogram_quantile(0.95, sum(rate(src_batch_change_agent_wake_duration_seconds_bucket{op="Handle"}[5m])) by (le)) +``` +
+ +
+ +#### batch-change-agents: batch_change_agent_wake_error_rate + +

Wake processing error rate over 5m

+ +Percentage of wake handler invocations that ended in an error. Wake failures are retried and rarely user-visible, but a sustained nonzero rate means threads are not being re-activated when new inbox items arrive. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100310` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +sum(increase(src_batch_change_agent_wake_errors_total{op="Handle"}[5m])) / (sum(increase(src_batch_change_agent_wake_total{op="Handle"}[5m])) + sum(increase(src_batch_change_agent_wake_errors_total{op="Handle"}[5m]))) * 100 +``` +
+ +
+ +### Batch Change Agents: Job queue stores + +#### batch-change-agents: batch_change_agent_store_operation_rate + +

Worker store operations per second by queue

+ +Rate of successful worker store operations (dequeue, mark-complete, etc.) for the batch change agent message and wake queues. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100400` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +sum by (domain) (rate(src_workerutil_dbworker_store_total{domain=~"^batch_change_agent.*_worker_store$"}[5m])) +``` +
+ +
+ +#### batch-change-agents: batch_change_agent_store_error_rate + +

Worker store error rate over 5m by queue

+ +Percentage of worker store operations that errored, per queue. A sustained nonzero rate usually points at database problems (contention, connection exhaustion, or migration drift) rather than the agent logic itself. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100401` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +sum by (domain) (increase(src_workerutil_dbworker_store_errors_total{domain=~"^batch_change_agent.*_worker_store$"}[5m])) / (sum by (domain) (increase(src_workerutil_dbworker_store_total{domain=~"^batch_change_agent.*_worker_store$"}[5m])) + sum by (domain) (increase(src_workerutil_dbworker_store_errors_total{domain=~"^batch_change_agent.*_worker_store$"}[5m]))) * 100 +``` +
+ +
+ +#### batch-change-agents: batch_change_agent_store_p95_duration + +

95th percentile worker store operation duration by queue

+ +95th percentile duration of worker store operations, per queue. Elevated values indicate database latency affecting how quickly jobs are dequeued and finalized. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100402` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +histogram_quantile(0.95, sum by (le, domain) (rate(src_workerutil_dbworker_store_duration_seconds_bucket{domain=~"^batch_change_agent.*_worker_store$"}[5m]))) +``` +
+ +
+ +### Batch Change Agents: Stalled job resetters + +#### batch-change-agents: batch_change_agent_records_reset + +

Stalled jobs reset to queued over 1h by queue

+ +Number of stalled batch change agent jobs that the resetter moved back to the queued state, per queue. + +Refer to the [alerts reference](alerts#batch-change-agents-batch_change_agent_records_reset) for 1 alert related to this panel. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100500` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +sum by (domain) (increase(src_dbworker_resetter_record_resets_total{domain=~"^batch_change_agent_.*_resetter$"}[1h])) +``` +
+ +
+ +#### batch-change-agents: batch_change_agent_record_reset_failures + +

Stalled jobs marked failed over 1h by queue

+ +Number of stalled jobs that exhausted their retries and were marked as failed, per queue. A nonzero value means some agent messages or wakes were permanently dropped after repeated crashes. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100501` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +sum by (domain) (increase(src_dbworker_resetter_record_reset_failures_total{domain=~"^batch_change_agent_.*_resetter$"}[1h])) +``` +
+ +
+ +#### batch-change-agents: batch_change_agent_stall_duration + +

95th percentile job stall duration before reset

+ +95th percentile time jobs spent stalled before the resetter recovered them, per queue. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100502` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +histogram_quantile(0.95, sum by (le, domain) (rate(src_dbworker_resetter_stall_duration_seconds_bucket{domain=~"^batch_change_agent_.*_resetter$"}[1h]))) +``` +
+ +
+ +### Batch Change Agents: Event streams (Redis pub/sub) + +#### batch-change-agents: batch_change_agent_message_stream_publish_rate + +

Message stream events published per second by type

+ +Rate of per-message stream events (turn deltas, tool calls, completion, etc.) successfully published to Redis for live delivery to connected clients. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100600` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +sum by (event_type) (rate(src_batch_change_agents_stream_publish_success_total[5m])) +``` +
+ +
+ +#### batch-change-agents: batch_change_agent_message_stream_publish_errors + +

Message stream publish errors per second

+ +Rate of per-message stream events that could not be published to Redis after retries. + +Refer to the [alerts reference](alerts#batch-change-agents-batch_change_agent_message_stream_publish_errors) for 1 alert related to this panel. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100601` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +sum(rate(src_batch_change_agents_stream_publish_errors_total[5m])) +``` +
+ +
+ +#### batch-change-agents: batch_change_agent_message_stream_transport_errors + +

Message stream transport errors per second

+ +Rate of Redis subscription transport errors (a subscription died mid-stream). Subscribers reconnect, but clients may briefly stop receiving live updates when this is elevated. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100602` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +sum(rate(src_batch_change_agents_stream_transport_errors_total[5m])) +``` +
+ +
+ +#### batch-change-agents: batch_change_agent_thread_stream_publish_rate + +

Thread stream events published per second by type

+ +Rate of thread-level stream events (new message created, title updated) successfully published to Redis. These prompt clients to refetch the thread. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100610` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +sum by (event_type) (rate(src_batch_change_agents_thread_stream_publish_success_total[5m])) +``` +
+ +
+ +#### batch-change-agents: batch_change_agent_thread_stream_publish_errors + +

Thread stream publish errors per second

+ +Rate of thread-level stream events that could not be published to Redis after retries. Clients recover thread state on their next refetch. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100611` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +sum(rate(src_batch_change_agents_thread_stream_publish_errors_total[5m])) +``` +
+ +
+ +#### batch-change-agents: batch_change_agent_thread_stream_transport_errors + +

Thread stream transport errors per second

+ +Rate of Redis subscription transport errors on the thread-level stream. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/batch-change-agents/batch-change-agents?viewPanel=100612` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Plane team.* + +
+Technical details + +Query: + +``` +sum(rate(src_batch_change_agents_thread_stream_transport_errors_total[5m])) +``` +
+ +
+ ## Sourcegraph external API

Monitoring for the Sourcegraph external API.