diff --git a/docs/admin/code-hosts/aws-codecommit.mdx b/docs/admin/code-hosts/aws-codecommit.mdx index b7e48bdce..c17a0986b 100644 --- a/docs/admin/code-hosts/aws-codecommit.mdx +++ b/docs/admin/code-hosts/aws-codecommit.mdx @@ -37,7 +37,7 @@ AWS CodeCommit connections support the following configuration options, which ar {/* SCHEMA_SYNC_START: admin/code_hosts/aws_codecommit.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-01-19T07:07:22Z */} +{/* Last updated: 2026-01-20T15:55:56Z */} ```json { // REQUIRED: diff --git a/docs/admin/code-hosts/azuredevops.mdx b/docs/admin/code-hosts/azuredevops.mdx index 36640c1d1..1ea266162 100644 --- a/docs/admin/code-hosts/azuredevops.mdx +++ b/docs/admin/code-hosts/azuredevops.mdx @@ -65,7 +65,7 @@ Azure DevOps connections support the following configuration options, which are {/* SCHEMA_SYNC_START: admin/code_hosts/azuredevops.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-01-19T07:07:22Z */} +{/* Last updated: 2026-01-20T15:55:56Z */} ```json // Authentication alternatives: token OR windowsPassword diff --git a/docs/admin/code-hosts/bitbucket-cloud.mdx b/docs/admin/code-hosts/bitbucket-cloud.mdx index 8b4e78786..64bea1831 100644 --- a/docs/admin/code-hosts/bitbucket-cloud.mdx +++ b/docs/admin/code-hosts/bitbucket-cloud.mdx @@ -116,7 +116,7 @@ Bitbucket Cloud connections support the following configuration options, which a {/* SCHEMA_SYNC_START: admin/code_hosts/bitbucket_cloud.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-01-19T07:07:22Z */} +{/* Last updated: 2026-01-20T15:55:56Z */} ```json { // The workspace access token to use when authenticating with Bitbucket Cloud. diff --git a/docs/admin/code-hosts/bitbucket-server.mdx b/docs/admin/code-hosts/bitbucket-server.mdx index 0f3cb9276..40a8846ad 100644 --- a/docs/admin/code-hosts/bitbucket-server.mdx +++ b/docs/admin/code-hosts/bitbucket-server.mdx @@ -202,7 +202,7 @@ Bitbucket Server / Bitbucket Data Center connections support the following confi {/* SCHEMA_SYNC_START: admin/code_hosts/bitbucket_server.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-01-19T07:07:22Z */} +{/* Last updated: 2026-01-20T15:55:56Z */} ```json // Authentication alternatives: token OR password diff --git a/docs/admin/code-hosts/gerrit.mdx b/docs/admin/code-hosts/gerrit.mdx index 3adae9730..845a57d47 100644 --- a/docs/admin/code-hosts/gerrit.mdx +++ b/docs/admin/code-hosts/gerrit.mdx @@ -113,7 +113,7 @@ Gerrit connections support the following configuration options, which are specif {/* SCHEMA_SYNC_START: admin/code_hosts/gerrit.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-01-19T07:07:22Z */} +{/* Last updated: 2026-01-20T15:55:56Z */} ```json { // If non-null, enforces Gerrit repository permissions. This requires that there is an item in the [site configuration json](https://sourcegraph.com/docs/admin/config/site_config#auth-providers) `auth.providers` field, of type "gerrit" with the same `url` field as specified in this `GerritConnection`. diff --git a/docs/admin/code-hosts/github.mdx b/docs/admin/code-hosts/github.mdx index 06e9a32a6..750fda838 100644 --- a/docs/admin/code-hosts/github.mdx +++ b/docs/admin/code-hosts/github.mdx @@ -454,7 +454,7 @@ GitHub connections support the following configuration options, which are specif {/* SCHEMA_SYNC_START: admin/code_hosts/github.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-01-19T07:07:22Z */} +{/* Last updated: 2026-01-20T15:55:56Z */} ```json // Authentication alternatives: token OR gitHubAppDetails OR externalAccount OR useRandomExternalAccount diff --git a/docs/admin/code-hosts/gitlab.mdx b/docs/admin/code-hosts/gitlab.mdx index ddfe360f3..b3ff79569 100644 --- a/docs/admin/code-hosts/gitlab.mdx +++ b/docs/admin/code-hosts/gitlab.mdx @@ -189,7 +189,7 @@ See [Internal rate limits](/admin/code-hosts/rate-limits#internal-rate-limits). {/* SCHEMA_SYNC_START: admin/code_hosts/gitlab.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-01-19T07:07:22Z */} +{/* Last updated: 2026-01-20T15:55:56Z */} ```json { // If non-null, enforces GitLab repository permissions. This requires that there be an item in the `auth.providers` field of type "gitlab" with the same `url` field as specified in this `GitLabConnection`. diff --git a/docs/admin/code-hosts/gitolite.mdx b/docs/admin/code-hosts/gitolite.mdx index c0a86c837..a7a1aed7d 100644 --- a/docs/admin/code-hosts/gitolite.mdx +++ b/docs/admin/code-hosts/gitolite.mdx @@ -25,7 +25,7 @@ To connect Gitolite to Sourcegraph: {/* SCHEMA_SYNC_START: admin/code_hosts/gitolite.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-01-19T07:07:22Z */} +{/* Last updated: 2026-01-20T15:55:56Z */} ```json { // A list of repositories to never mirror from this Gitolite instance. Supports excluding by exact name ({"name": "foo"}). diff --git a/docs/admin/code-hosts/other.mdx b/docs/admin/code-hosts/other.mdx index 04db2cfe9..58ec1f7bf 100644 --- a/docs/admin/code-hosts/other.mdx +++ b/docs/admin/code-hosts/other.mdx @@ -68,7 +68,7 @@ Repositories must be listed individually: {/* SCHEMA_SYNC_START: admin/code_hosts/other_external_service.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-01-19T07:07:22Z */} +{/* Last updated: 2026-01-20T15:55:56Z */} ```json { // A list of repositories to never mirror by name after applying repositoryPathPattern. Supports excluding by exact name ({"name": "myrepo"}) or regular expression ({"pattern": ".*secret.*"}). diff --git a/docs/admin/code-hosts/phabricator.mdx b/docs/admin/code-hosts/phabricator.mdx index 2309fdad6..1f6a0632f 100644 --- a/docs/admin/code-hosts/phabricator.mdx +++ b/docs/admin/code-hosts/phabricator.mdx @@ -76,7 +76,7 @@ The Sourcegraph instance's site admin must [update the `corsOrigin` site config {/* SCHEMA_SYNC_START: admin/code_hosts/phabricator.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-01-19T07:07:22Z */} +{/* Last updated: 2026-01-20T15:55:56Z */} ```json { // SSH cipher to use when cloning via SSH. Must be a valid choice from `ssh -Q cipher`. diff --git a/docs/admin/config/settings.mdx b/docs/admin/config/settings.mdx index 9d998927e..59337a534 100644 --- a/docs/admin/config/settings.mdx +++ b/docs/admin/config/settings.mdx @@ -27,7 +27,7 @@ Settings options and their default values are shown below. {/* SCHEMA_SYNC_START: admin/config/settings.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-01-19T07:07:22Z */} +{/* Last updated: 2026-01-20T15:55:56Z */} ```json { diff --git a/docs/admin/config/site-config.mdx b/docs/admin/config/site-config.mdx index edfbc04b6..06ef8da6a 100644 --- a/docs/admin/config/site-config.mdx +++ b/docs/admin/config/site-config.mdx @@ -21,7 +21,7 @@ All site configuration options and their default values are shown below. {/* SCHEMA_SYNC_START: admin/config/site.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-01-19T07:07:22Z */} +{/* Last updated: 2026-01-20T15:55:56Z */} ```json { diff --git a/docs/admin/repo/perforce.mdx b/docs/admin/repo/perforce.mdx index 59637d5b9..b9ea1daeb 100644 --- a/docs/admin/repo/perforce.mdx +++ b/docs/admin/repo/perforce.mdx @@ -228,7 +228,7 @@ With this setting, Sourcegraph will ignore any rules with a host other than `*`, {/* SCHEMA_SYNC_START: admin/code_hosts/perforce.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-01-19T07:07:22Z */} +{/* Last updated: 2026-01-20T15:55:56Z */} ```json { // If non-null, enforces Perforce depot permissions. diff --git a/docs/self-hosted/observability/alerts.mdx b/docs/self-hosted/observability/alerts.mdx index ee1bc9665..d3b192e32 100644 --- a/docs/self-hosted/observability/alerts.mdx +++ b/docs/self-hosted/observability/alerts.mdx @@ -2494,693 +2494,24 @@ Generated query for critical alert: `max((sum by (app_name, db_name) (increase(s
-## precise-code-intel-worker: cpu_usage_percentage - -

CPU usage

- -**Descriptions** - -- warning precise-code-intel-worker: 95%+ CPU usage for 10m0s - -**Next steps** - -- Consider increasing CPU limits or scaling out. -- Learn more about the related dashboard panel in the [dashboards reference](dashboards#precise-code-intel-worker-cpu_usage_percentage). -- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: - -```json -"observability.silenceAlerts": [ - "warning_precise-code-intel-worker_cpu_usage_percentage" -] -``` - -*Managed by the Sourcegraph Code Understanding team.* - -
-Technical details - -Generated query for warning alert: `max((cadvisor_container_cpu_usage_percentage_total{name=~"^precise-code-intel-worker.*"}) >= 95)` - -
- -
- -## precise-code-intel-worker: memory_rss - -

memory (RSS)

- -**Descriptions** - -- warning precise-code-intel-worker: 90%+ memory (RSS) for 10m0s - -**Next steps** - -- Consider increasing memory limits or scaling out. -- More help interpreting this metric is available in the [dashboards reference](dashboards#precise-code-intel-worker-memory_rss). -- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: - -```json -"observability.silenceAlerts": [ - "warning_precise-code-intel-worker_memory_rss" -] -``` - -*Managed by the Sourcegraph Code Understanding team.* - -
-Technical details - -Generated query for warning alert: `max((max by (name) (container_memory_rss{name=~"^precise-code-intel-worker.*"\} / container_spec_memory_limit_bytes\{name=~"^precise-code-intel-worker.*"}) * 100) >= 90)` - -
- -
- -## precise-code-intel-worker: container_cpu_usage - -

container cpu usage total (1m average) across all cores by instance

- -**Descriptions** - -- warning precise-code-intel-worker: 99%+ container cpu usage total (1m average) across all cores by instance - -**Next steps** - -- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`. -- **Docker Compose:** Consider increasing `cpus:` of the precise-code-intel-worker container in `docker-compose.yml`. -- Learn more about the related dashboard panel in the [dashboards reference](dashboards#precise-code-intel-worker-container_cpu_usage). -- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: - -```json -"observability.silenceAlerts": [ - "warning_precise-code-intel-worker_container_cpu_usage" -] -``` - -*Managed by the Sourcegraph Code Understanding team.* - -
-Technical details - -Generated query for warning alert: `max((cadvisor_container_cpu_usage_percentage_total{name=~"^precise-code-intel-worker.*"}) >= 99)` - -
- -
- -## precise-code-intel-worker: container_memory_usage - -

container memory usage by instance

- -**Descriptions** - -- warning precise-code-intel-worker: 99%+ container memory usage by instance - -**Next steps** - -- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. -- **Docker Compose:** Consider increasing `memory:` of precise-code-intel-worker container in `docker-compose.yml`. -- Learn more about the related dashboard panel in the [dashboards reference](dashboards#precise-code-intel-worker-container_memory_usage). -- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: - -```json -"observability.silenceAlerts": [ - "warning_precise-code-intel-worker_container_memory_usage" -] -``` - -*Managed by the Sourcegraph Code Understanding team.* - -
-Technical details - -Generated query for warning alert: `max((cadvisor_container_memory_usage_percentage_total{name=~"^precise-code-intel-worker.*"}) >= 99)` - -
- -
- -## precise-code-intel-worker: provisioning_container_cpu_usage_long_term - -

container cpu usage total (90th percentile over 1d) across all cores by instance

- -**Descriptions** - -- warning precise-code-intel-worker: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s - -**Next steps** - -- **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the precise-code-intel-worker service. -- **Docker Compose:** Consider increasing `cpus:` of the precise-code-intel-worker container in `docker-compose.yml`. -- Learn more about the related dashboard panel in the [dashboards reference](dashboards#precise-code-intel-worker-provisioning_container_cpu_usage_long_term). -- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: - -```json -"observability.silenceAlerts": [ - "warning_precise-code-intel-worker_provisioning_container_cpu_usage_long_term" -] -``` - -*Managed by the Sourcegraph Code Understanding team.* - -
-Technical details - -Generated query for warning alert: `max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^precise-code-intel-worker.*"}[1d])) >= 80)` - -
- -
- -## precise-code-intel-worker: provisioning_container_memory_usage_long_term - -

container memory usage (1d maximum) by instance

- -**Descriptions** - -- warning precise-code-intel-worker: 80%+ container memory usage (1d maximum) by instance for 336h0m0s - -**Next steps** - -- **Kubernetes:** Consider increasing memory limits in the `Deployment.yaml` for the precise-code-intel-worker service. -- **Docker Compose:** Consider increasing `memory:` of the precise-code-intel-worker container in `docker-compose.yml`. -- Learn more about the related dashboard panel in the [dashboards reference](dashboards#precise-code-intel-worker-provisioning_container_memory_usage_long_term). -- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: - -```json -"observability.silenceAlerts": [ - "warning_precise-code-intel-worker_provisioning_container_memory_usage_long_term" -] -``` - -*Managed by the Sourcegraph Code Understanding team.* - -
-Technical details - -Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^precise-code-intel-worker.*"}[1d])) >= 80)` - -
- -
- -## precise-code-intel-worker: provisioning_container_cpu_usage_short_term - -

container cpu usage total (5m maximum) across all cores by instance

- -**Descriptions** - -- warning precise-code-intel-worker: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s - -**Next steps** - -- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`. -- **Docker Compose:** Consider increasing `cpus:` of the precise-code-intel-worker container in `docker-compose.yml`. -- Learn more about the related dashboard panel in the [dashboards reference](dashboards#precise-code-intel-worker-provisioning_container_cpu_usage_short_term). -- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: - -```json -"observability.silenceAlerts": [ - "warning_precise-code-intel-worker_provisioning_container_cpu_usage_short_term" -] -``` - -*Managed by the Sourcegraph Code Understanding team.* - -
-Technical details - -Generated query for warning alert: `max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^precise-code-intel-worker.*"}[5m])) >= 90)` - -
- -
- -## precise-code-intel-worker: provisioning_container_memory_usage_short_term - -

container memory usage (5m maximum) by instance

- -**Descriptions** - -- warning precise-code-intel-worker: 90%+ container memory usage (5m maximum) by instance - -**Next steps** - -- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. -- **Docker Compose:** Consider increasing `memory:` of precise-code-intel-worker container in `docker-compose.yml`. -- Learn more about the related dashboard panel in the [dashboards reference](dashboards#precise-code-intel-worker-provisioning_container_memory_usage_short_term). -- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: - -```json -"observability.silenceAlerts": [ - "warning_precise-code-intel-worker_provisioning_container_memory_usage_short_term" -] -``` - -*Managed by the Sourcegraph Code Understanding team.* - -
-Technical details - -Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^precise-code-intel-worker.*"}[5m])) >= 90)` - -
- -
- -## precise-code-intel-worker: container_oomkill_events_total - -

container OOMKILL events total by instance

- -**Descriptions** - -- warning precise-code-intel-worker: 1+ container OOMKILL events total by instance - -**Next steps** - -- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. -- **Docker Compose:** Consider increasing `memory:` of precise-code-intel-worker container in `docker-compose.yml`. -- More help interpreting this metric is available in the [dashboards reference](dashboards#precise-code-intel-worker-container_oomkill_events_total). -- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: - -```json -"observability.silenceAlerts": [ - "warning_precise-code-intel-worker_container_oomkill_events_total" -] -``` - -*Managed by the Sourcegraph Code Understanding team.* - -
-Technical details - -Generated query for warning alert: `max((max by (name) (container_oom_events_total{name=~"^precise-code-intel-worker.*"})) >= 1)` - -
- -
- -## precise-code-intel-worker: go_goroutines - -

maximum active goroutines

- -**Descriptions** - -- warning precise-code-intel-worker: 10000+ maximum active goroutines for 10m0s - -**Next steps** - -- More help interpreting this metric is available in the [dashboards reference](dashboards#precise-code-intel-worker-go_goroutines). -- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: - -```json -"observability.silenceAlerts": [ - "warning_precise-code-intel-worker_go_goroutines" -] -``` - -*Managed by the Sourcegraph Code Understanding team.* - -
-Technical details - -Generated query for warning alert: `max((max by (instance) (go_goroutines{job=~".*precise-code-intel-worker"})) >= 10000)` - -
- -
- -## precise-code-intel-worker: go_gc_duration_seconds - -

maximum go garbage collection duration

- -**Descriptions** - -- warning precise-code-intel-worker: 2s+ maximum go garbage collection duration - -**Next steps** - -- Learn more about the related dashboard panel in the [dashboards reference](dashboards#precise-code-intel-worker-go_gc_duration_seconds). -- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: - -```json -"observability.silenceAlerts": [ - "warning_precise-code-intel-worker_go_gc_duration_seconds" -] -``` - -*Managed by the Sourcegraph Code Understanding team.* - -
-Technical details - -Generated query for warning alert: `max((max by (instance) (go_gc_duration_seconds{job=~".*precise-code-intel-worker"})) >= 2)` - -
- -
- -## precise-code-intel-worker: pods_available_percentage - -

percentage pods available

- -**Descriptions** - -- critical precise-code-intel-worker: less than 90% percentage pods available for 10m0s - -**Next steps** - -- Determine if the pod was OOM killed using `kubectl describe pod precise-code-intel-worker` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`. -- Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p precise-code-intel-worker`. -- Learn more about the related dashboard panel in the [dashboards reference](dashboards#precise-code-intel-worker-pods_available_percentage). -- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: - -```json -"observability.silenceAlerts": [ - "critical_precise-code-intel-worker_pods_available_percentage" -] -``` - -*Managed by the Sourcegraph Code Understanding team.* - -
-Technical details - -Generated query for critical alert: `min((sum by (app) (up{app=~".*precise-code-intel-worker"\}) / count by (app) (up\{app=~".*precise-code-intel-worker"}) * 100) <= 90)` - -
- -
- -## syntactic-indexing: mean_blocked_seconds_per_conn_request - -

mean blocked seconds per conn request

- -**Descriptions** - -- warning syntactic-indexing: 0.1s+ mean blocked seconds per conn request for 10m0s -- critical syntactic-indexing: 0.5s+ mean blocked seconds per conn request for 10m0s - -**Next steps** - -- Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed -- Scale up Postgres memory/cpus - [see our scaling guide](https://sourcegraph.com/docs/admin/config/postgres-conf) -- If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights -- Learn more about the related dashboard panel in the [dashboards reference](dashboards#syntactic-indexing-mean_blocked_seconds_per_conn_request). -- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: - -```json -"observability.silenceAlerts": [ - "warning_syntactic-indexing_mean_blocked_seconds_per_conn_request", - "critical_syntactic-indexing_mean_blocked_seconds_per_conn_request" -] -``` - -*Managed by the Sourcegraph Platform team.* - -
-Technical details - -Generated query for warning alert: `max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="syntactic-code-intel-worker"\}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for\{app_name="syntactic-code-intel-worker"}[5m]))) >= 0.1)` - -Generated query for critical alert: `max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="syntactic-code-intel-worker"\}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for\{app_name="syntactic-code-intel-worker"}[5m]))) >= 0.5)` - -
- -
- -## syntactic-indexing: cpu_usage_percentage - -

CPU usage

- -**Descriptions** - -- warning syntactic-indexing: 95%+ CPU usage for 10m0s - -**Next steps** - -- Consider increasing CPU limits or scaling out. -- Learn more about the related dashboard panel in the [dashboards reference](dashboards#syntactic-indexing-cpu_usage_percentage). -- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: - -```json -"observability.silenceAlerts": [ - "warning_syntactic-indexing_cpu_usage_percentage" -] -``` - -*Managed by the Sourcegraph Code Understanding team.* - -
-Technical details - -Generated query for warning alert: `max((cadvisor_container_cpu_usage_percentage_total{name=~"^syntactic-code-intel-worker.*"}) >= 95)` - -
- -
- -## syntactic-indexing: memory_rss - -

memory (RSS)

- -**Descriptions** - -- warning syntactic-indexing: 90%+ memory (RSS) for 10m0s - -**Next steps** - -- Consider increasing memory limits or scaling out. -- More help interpreting this metric is available in the [dashboards reference](dashboards#syntactic-indexing-memory_rss). -- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: - -```json -"observability.silenceAlerts": [ - "warning_syntactic-indexing_memory_rss" -] -``` - -*Managed by the Sourcegraph Code Understanding team.* - -
-Technical details - -Generated query for warning alert: `max((max by (name) (container_memory_rss{name=~"^syntactic-code-intel-worker.*"\} / container_spec_memory_limit_bytes\{name=~"^syntactic-code-intel-worker.*"}) * 100) >= 90)` - -
- -
- -## syntactic-indexing: container_cpu_usage - -

container cpu usage total (1m average) across all cores by instance

- -**Descriptions** - -- warning syntactic-indexing: 99%+ container cpu usage total (1m average) across all cores by instance - -**Next steps** - -- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`. -- **Docker Compose:** Consider increasing `cpus:` of the syntactic-code-intel-worker container in `docker-compose.yml`. -- Learn more about the related dashboard panel in the [dashboards reference](dashboards#syntactic-indexing-container_cpu_usage). -- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: - -```json -"observability.silenceAlerts": [ - "warning_syntactic-indexing_container_cpu_usage" -] -``` - -*Managed by the Sourcegraph Code Understanding team.* - -
-Technical details - -Generated query for warning alert: `max((cadvisor_container_cpu_usage_percentage_total{name=~"^syntactic-code-intel-worker.*"}) >= 99)` - -
- -
- -## syntactic-indexing: container_memory_usage - -

container memory usage by instance

- -**Descriptions** - -- warning syntactic-indexing: 99%+ container memory usage by instance - -**Next steps** - -- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. -- **Docker Compose:** Consider increasing `memory:` of syntactic-code-intel-worker container in `docker-compose.yml`. -- Learn more about the related dashboard panel in the [dashboards reference](dashboards#syntactic-indexing-container_memory_usage). -- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: - -```json -"observability.silenceAlerts": [ - "warning_syntactic-indexing_container_memory_usage" -] -``` - -*Managed by the Sourcegraph Code Understanding team.* - -
-Technical details - -Generated query for warning alert: `max((cadvisor_container_memory_usage_percentage_total{name=~"^syntactic-code-intel-worker.*"}) >= 99)` - -
- -
- -## syntactic-indexing: provisioning_container_cpu_usage_long_term - -

container cpu usage total (90th percentile over 1d) across all cores by instance

- -**Descriptions** - -- warning syntactic-indexing: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s - -**Next steps** - -- **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the syntactic-code-intel-worker service. -- **Docker Compose:** Consider increasing `cpus:` of the syntactic-code-intel-worker container in `docker-compose.yml`. -- Learn more about the related dashboard panel in the [dashboards reference](dashboards#syntactic-indexing-provisioning_container_cpu_usage_long_term). -- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: - -```json -"observability.silenceAlerts": [ - "warning_syntactic-indexing_provisioning_container_cpu_usage_long_term" -] -``` - -*Managed by the Sourcegraph Code Understanding team.* - -
-Technical details - -Generated query for warning alert: `max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^syntactic-code-intel-worker.*"}[1d])) >= 80)` - -
- -
- -## syntactic-indexing: provisioning_container_memory_usage_long_term - -

container memory usage (1d maximum) by instance

- -**Descriptions** - -- warning syntactic-indexing: 80%+ container memory usage (1d maximum) by instance for 336h0m0s - -**Next steps** - -- **Kubernetes:** Consider increasing memory limits in the `Deployment.yaml` for the syntactic-code-intel-worker service. -- **Docker Compose:** Consider increasing `memory:` of the syntactic-code-intel-worker container in `docker-compose.yml`. -- Learn more about the related dashboard panel in the [dashboards reference](dashboards#syntactic-indexing-provisioning_container_memory_usage_long_term). -- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: - -```json -"observability.silenceAlerts": [ - "warning_syntactic-indexing_provisioning_container_memory_usage_long_term" -] -``` - -*Managed by the Sourcegraph Code Understanding team.* - -
-Technical details - -Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^syntactic-code-intel-worker.*"}[1d])) >= 80)` - -
- -
- -## syntactic-indexing: provisioning_container_cpu_usage_short_term - -

container cpu usage total (5m maximum) across all cores by instance

- -**Descriptions** - -- warning syntactic-indexing: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s - -**Next steps** - -- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`. -- **Docker Compose:** Consider increasing `cpus:` of the syntactic-code-intel-worker container in `docker-compose.yml`. -- Learn more about the related dashboard panel in the [dashboards reference](dashboards#syntactic-indexing-provisioning_container_cpu_usage_short_term). -- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: - -```json -"observability.silenceAlerts": [ - "warning_syntactic-indexing_provisioning_container_cpu_usage_short_term" -] -``` - -*Managed by the Sourcegraph Code Understanding team.* - -
-Technical details - -Generated query for warning alert: `max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^syntactic-code-intel-worker.*"}[5m])) >= 90)` - -
- -
- -## syntactic-indexing: provisioning_container_memory_usage_short_term - -

container memory usage (5m maximum) by instance

- -**Descriptions** - -- warning syntactic-indexing: 90%+ container memory usage (5m maximum) by instance - -**Next steps** - -- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. -- **Docker Compose:** Consider increasing `memory:` of syntactic-code-intel-worker container in `docker-compose.yml`. -- Learn more about the related dashboard panel in the [dashboards reference](dashboards#syntactic-indexing-provisioning_container_memory_usage_short_term). -- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: - -```json -"observability.silenceAlerts": [ - "warning_syntactic-indexing_provisioning_container_memory_usage_short_term" -] -``` - -*Managed by the Sourcegraph Code Understanding team.* - -
-Technical details - -Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^syntactic-code-intel-worker.*"}[5m])) >= 90)` - -
- -
- -## syntactic-indexing: container_oomkill_events_total +## precise-code-intel-worker: pods_available_percentage -

container OOMKILL events total by instance

+

percentage pods available

**Descriptions** -- warning syntactic-indexing: 1+ container OOMKILL events total by instance +- critical precise-code-intel-worker: less than 90% percentage pods available for 10m0s **Next steps** -- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. -- **Docker Compose:** Consider increasing `memory:` of syntactic-code-intel-worker container in `docker-compose.yml`. -- More help interpreting this metric is available in the [dashboards reference](dashboards#syntactic-indexing-container_oomkill_events_total). +- Determine if the pod was OOM killed using `kubectl describe pod precise-code-intel-worker` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`. +- Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p precise-code-intel-worker`. +- Learn more about the related dashboard panel in the [dashboards reference](dashboards#precise-code-intel-worker-pods_available_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ - "warning_syntactic-indexing_container_oomkill_events_total" + "critical_precise-code-intel-worker_pods_available_percentage" ] ``` @@ -3189,67 +2520,44 @@ Generated query for warning alert: `max((max_over_time(cadvisor_container_memory
Technical details -Generated query for warning alert: `max((max by (name) (container_oom_events_total{name=~"^syntactic-code-intel-worker.*"})) >= 1)` +Generated query for critical alert: `min((sum by (app) (up{app=~".*precise-code-intel-worker"\}) / count by (app) (up\{app=~".*precise-code-intel-worker"}) * 100) <= 90)`

-## syntactic-indexing: go_goroutines +## syntactic-indexing: mean_blocked_seconds_per_conn_request -

maximum active goroutines

+

mean blocked seconds per conn request

**Descriptions** -- warning syntactic-indexing: 10000+ maximum active goroutines for 10m0s +- warning syntactic-indexing: 0.1s+ mean blocked seconds per conn request for 10m0s +- critical syntactic-indexing: 0.5s+ mean blocked seconds per conn request for 10m0s **Next steps** -- More help interpreting this metric is available in the [dashboards reference](dashboards#syntactic-indexing-go_goroutines). +- Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed +- Scale up Postgres memory/cpus - [see our scaling guide](https://sourcegraph.com/docs/admin/config/postgres-conf) +- If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights +- Learn more about the related dashboard panel in the [dashboards reference](dashboards#syntactic-indexing-mean_blocked_seconds_per_conn_request). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ - "warning_syntactic-indexing_go_goroutines" + "warning_syntactic-indexing_mean_blocked_seconds_per_conn_request", + "critical_syntactic-indexing_mean_blocked_seconds_per_conn_request" ] ``` -*Managed by the Sourcegraph Code Understanding team.* +*Managed by the Sourcegraph Platform team.*
Technical details -Generated query for warning alert: `max((max by (instance) (go_goroutines{job=~".*syntactic-code-intel-worker"})) >= 10000)` - -
- -
- -## syntactic-indexing: go_gc_duration_seconds - -

maximum go garbage collection duration

- -**Descriptions** - -- warning syntactic-indexing: 2s+ maximum go garbage collection duration - -**Next steps** - -- Learn more about the related dashboard panel in the [dashboards reference](dashboards#syntactic-indexing-go_gc_duration_seconds). -- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: - -```json -"observability.silenceAlerts": [ - "warning_syntactic-indexing_go_gc_duration_seconds" -] -``` - -*Managed by the Sourcegraph Code Understanding team.* - -
-Technical details +Generated query for warning alert: `max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="syntactic-code-intel-worker"\}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for\{app_name="syntactic-code-intel-worker"}[5m]))) >= 0.1)` -Generated query for warning alert: `max((max by (instance) (go_gc_duration_seconds{job=~".*syntactic-code-intel-worker"})) >= 2)` +Generated query for critical alert: `max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="syntactic-code-intel-worker"\}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for\{app_name="syntactic-code-intel-worker"}[5m]))) >= 0.5)`
diff --git a/docs/self-hosted/observability/dashboards.mdx b/docs/self-hosted/observability/dashboards.mdx index f9fce4367..52dd7d12a 100644 --- a/docs/self-hosted/observability/dashboards.mdx +++ b/docs/self-hosted/observability/dashboards.mdx @@ -12214,7 +12214,7 @@ sum by (app_name, db_name) (increase(src_pgsql_conns_closed_max_idle_time{app_na

CPU usage

-Refer to the [alerts reference](alerts#precise-code-intel-worker-cpu_usage_percentage) for 1 alert related to this panel. +This panel has no related alerts. To see this panel, visit `/-/debug/grafana/d/precise-code-intel-worker/precise-code-intel-worker?viewPanel=100700` on your Sourcegraph instance. @@ -12284,9 +12284,7 @@ max by (name) (container_memory_working_set_bytes{name=~"^precise-code-intel-wor

Memory (RSS)

-The total anonymous memory in use by the application, which includes Go stack and heap. This memory is is non-reclaimable, and high usage may trigger OOM kills. Note: the metric is named RSS because to match the cadvisor name, but `anonymous` is more accurate." - -Refer to the [alerts reference](alerts#precise-code-intel-worker-memory_rss) for 1 alert related to this panel. +This panel has no related alerts. To see this panel, visit `/-/debug/grafana/d/precise-code-intel-worker/precise-code-intel-worker?viewPanel=100710` on your Sourcegraph instance. @@ -12390,7 +12388,7 @@ count by(name) ((time() - container_last_seen{name=~"^precise-code-intel-worker.

Container cpu usage total (1m average) across all cores by instance

-Refer to the [alerts reference](alerts#precise-code-intel-worker-container_cpu_usage) for 1 alert related to this panel. +This panel has no related alerts. To see this panel, visit `/-/debug/grafana/d/precise-code-intel-worker/precise-code-intel-worker?viewPanel=100801` on your Sourcegraph instance. @@ -12412,7 +12410,7 @@ cadvisor_container_cpu_usage_percentage_total{name=~"^precise-code-intel-worker.

Container memory usage by instance

-Refer to the [alerts reference](alerts#precise-code-intel-worker-container_memory_usage) for 1 alert related to this panel. +This panel has no related alerts. To see this panel, visit `/-/debug/grafana/d/precise-code-intel-worker/precise-code-intel-worker?viewPanel=100802` on your Sourcegraph instance. @@ -12461,7 +12459,7 @@ sum by(name) (rate(container_fs_reads_total{name=~"^precise-code-intel-worker.*"

Container cpu usage total (90th percentile over 1d) across all cores by instance

-Refer to the [alerts reference](alerts#precise-code-intel-worker-provisioning_container_cpu_usage_long_term) for 1 alert related to this panel. +This panel has no related alerts. To see this panel, visit `/-/debug/grafana/d/precise-code-intel-worker/precise-code-intel-worker?viewPanel=100900` on your Sourcegraph instance. @@ -12483,7 +12481,7 @@ quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^pr

Container memory usage (1d maximum) by instance

-Refer to the [alerts reference](alerts#precise-code-intel-worker-provisioning_container_memory_usage_long_term) for 1 alert related to this panel. +This panel has no related alerts. To see this panel, visit `/-/debug/grafana/d/precise-code-intel-worker/precise-code-intel-worker?viewPanel=100901` on your Sourcegraph instance. @@ -12505,7 +12503,7 @@ max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^precise-c

Container cpu usage total (5m maximum) across all cores by instance

-Refer to the [alerts reference](alerts#precise-code-intel-worker-provisioning_container_cpu_usage_short_term) for 1 alert related to this panel. +This panel has no related alerts. To see this panel, visit `/-/debug/grafana/d/precise-code-intel-worker/precise-code-intel-worker?viewPanel=100910` on your Sourcegraph instance. @@ -12527,7 +12525,7 @@ max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^precise-code

Container memory usage (5m maximum) by instance

-Refer to the [alerts reference](alerts#precise-code-intel-worker-provisioning_container_memory_usage_short_term) for 1 alert related to this panel. +This panel has no related alerts. To see this panel, visit `/-/debug/grafana/d/precise-code-intel-worker/precise-code-intel-worker?viewPanel=100911` on your Sourcegraph instance. @@ -12549,10 +12547,7 @@ max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^precise-c

Container OOMKILL events total by instance

-This value indicates the total number of times the container main process or child processes were terminated by OOM killer. -When it occurs frequently, it is an indicator of underprovisioning. - -Refer to the [alerts reference](alerts#precise-code-intel-worker-container_oomkill_events_total) for 1 alert related to this panel. +This panel has no related alerts. To see this panel, visit `/-/debug/grafana/d/precise-code-intel-worker/precise-code-intel-worker?viewPanel=100912` on your Sourcegraph instance. @@ -12576,9 +12571,7 @@ max by (name) (container_oom_events_total{name=~"^precise-code-intel-worker.*"})

Maximum active goroutines

-A high value here indicates a possible goroutine leak. - -Refer to the [alerts reference](alerts#precise-code-intel-worker-go_goroutines) for 1 alert related to this panel. +This panel has no related alerts. To see this panel, visit `/-/debug/grafana/d/precise-code-intel-worker/precise-code-intel-worker?viewPanel=101000` on your Sourcegraph instance. @@ -12600,7 +12593,7 @@ max by(instance) (go_goroutines{job=~".*precise-code-intel-worker"})

Maximum go garbage collection duration

-Refer to the [alerts reference](alerts#precise-code-intel-worker-go_gc_duration_seconds) for 1 alert related to this panel. +This panel has no related alerts. To see this panel, visit `/-/debug/grafana/d/precise-code-intel-worker/precise-code-intel-worker?viewPanel=101001` on your Sourcegraph instance. @@ -13213,7 +13206,7 @@ sum by (app_name, db_name) (increase(src_pgsql_conns_closed_max_idle_time{app_na

CPU usage

-Refer to the [alerts reference](alerts#syntactic-indexing-cpu_usage_percentage) for 1 alert related to this panel. +This panel has no related alerts. To see this panel, visit `/-/debug/grafana/d/syntactic-indexing/syntactic-indexing?viewPanel=100400` on your Sourcegraph instance. @@ -13283,9 +13276,7 @@ max by (name) (container_memory_working_set_bytes{name=~"^syntactic-code-intel-w

Memory (RSS)

-The total anonymous memory in use by the application, which includes Go stack and heap. This memory is is non-reclaimable, and high usage may trigger OOM kills. Note: the metric is named RSS because to match the cadvisor name, but `anonymous` is more accurate." - -Refer to the [alerts reference](alerts#syntactic-indexing-memory_rss) for 1 alert related to this panel. +This panel has no related alerts. To see this panel, visit `/-/debug/grafana/d/syntactic-indexing/syntactic-indexing?viewPanel=100410` on your Sourcegraph instance. @@ -13389,7 +13380,7 @@ count by(name) ((time() - container_last_seen{name=~"^syntactic-code-intel-worke

Container cpu usage total (1m average) across all cores by instance

-Refer to the [alerts reference](alerts#syntactic-indexing-container_cpu_usage) for 1 alert related to this panel. +This panel has no related alerts. To see this panel, visit `/-/debug/grafana/d/syntactic-indexing/syntactic-indexing?viewPanel=100501` on your Sourcegraph instance. @@ -13411,7 +13402,7 @@ cadvisor_container_cpu_usage_percentage_total{name=~"^syntactic-code-intel-worke

Container memory usage by instance

-Refer to the [alerts reference](alerts#syntactic-indexing-container_memory_usage) for 1 alert related to this panel. +This panel has no related alerts. To see this panel, visit `/-/debug/grafana/d/syntactic-indexing/syntactic-indexing?viewPanel=100502` on your Sourcegraph instance. @@ -13460,7 +13451,7 @@ sum by(name) (rate(container_fs_reads_total{name=~"^syntactic-code-intel-worker.

Container cpu usage total (90th percentile over 1d) across all cores by instance

-Refer to the [alerts reference](alerts#syntactic-indexing-provisioning_container_cpu_usage_long_term) for 1 alert related to this panel. +This panel has no related alerts. To see this panel, visit `/-/debug/grafana/d/syntactic-indexing/syntactic-indexing?viewPanel=100600` on your Sourcegraph instance. @@ -13482,7 +13473,7 @@ quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^sy

Container memory usage (1d maximum) by instance

-Refer to the [alerts reference](alerts#syntactic-indexing-provisioning_container_memory_usage_long_term) for 1 alert related to this panel. +This panel has no related alerts. To see this panel, visit `/-/debug/grafana/d/syntactic-indexing/syntactic-indexing?viewPanel=100601` on your Sourcegraph instance. @@ -13504,7 +13495,7 @@ max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^syntactic

Container cpu usage total (5m maximum) across all cores by instance

-Refer to the [alerts reference](alerts#syntactic-indexing-provisioning_container_cpu_usage_short_term) for 1 alert related to this panel. +This panel has no related alerts. To see this panel, visit `/-/debug/grafana/d/syntactic-indexing/syntactic-indexing?viewPanel=100610` on your Sourcegraph instance. @@ -13526,7 +13517,7 @@ max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^syntactic-co

Container memory usage (5m maximum) by instance

-Refer to the [alerts reference](alerts#syntactic-indexing-provisioning_container_memory_usage_short_term) for 1 alert related to this panel. +This panel has no related alerts. To see this panel, visit `/-/debug/grafana/d/syntactic-indexing/syntactic-indexing?viewPanel=100611` on your Sourcegraph instance. @@ -13548,10 +13539,7 @@ max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^syntactic

Container OOMKILL events total by instance

-This value indicates the total number of times the container main process or child processes were terminated by OOM killer. -When it occurs frequently, it is an indicator of underprovisioning. - -Refer to the [alerts reference](alerts#syntactic-indexing-container_oomkill_events_total) for 1 alert related to this panel. +This panel has no related alerts. To see this panel, visit `/-/debug/grafana/d/syntactic-indexing/syntactic-indexing?viewPanel=100612` on your Sourcegraph instance. @@ -13575,9 +13563,7 @@ max by (name) (container_oom_events_total{name=~"^syntactic-code-intel-worker.*"

Maximum active goroutines

-A high value here indicates a possible goroutine leak. - -Refer to the [alerts reference](alerts#syntactic-indexing-go_goroutines) for 1 alert related to this panel. +This panel has no related alerts. To see this panel, visit `/-/debug/grafana/d/syntactic-indexing/syntactic-indexing?viewPanel=100700` on your Sourcegraph instance. @@ -13599,7 +13585,7 @@ max by(instance) (go_goroutines{job=~".*syntactic-code-intel-worker"})

Maximum go garbage collection duration

-Refer to the [alerts reference](alerts#syntactic-indexing-go_gc_duration_seconds) for 1 alert related to this panel. +This panel has no related alerts. To see this panel, visit `/-/debug/grafana/d/syntactic-indexing/syntactic-indexing?viewPanel=100701` on your Sourcegraph instance.