feat: ci-metrics data quality + dashboard improvements#20489
Open
ludamad wants to merge 7 commits intomerge-train/spartanfrom
Open
feat: ci-metrics data quality + dashboard improvements#20489ludamad wants to merge 7 commits intomerge-train/spartanfrom
ludamad wants to merge 7 commits intomerge-train/spartanfrom
Conversation
- Fix subprocess race condition with fcntl file lock - Warm billing caches on startup with --preload - Add test timings link to all dashboard nav bars - Reduce gunicorn workers from 100 to 50 - Add METRICS_DB_PATH env var for SQLite location - Fix Content-Encoding stripping for proxied responses - Kill stale ci-metrics process before restart
- Track test successes via daily aggregate table (test_daily_stats) without persisting individual passed events; backfill from existing test_events - Fix instance type detection in log_ci_run to prefer EC2_INSTANCE_TYPE env var over metadata endpoint (which fails in Docker) - Add CloudTrail backfill to resolve unknown instance types for historical CI runs and recalculate costs - Add test success counts to CI Insights chart (stacked bar: successes, flakes, failures) - Add time period metadata to all API responses and display in dashboard headers (ci-insights, cost-overview, test-timings) - Use test_daily_stats for CI performance endpoint counts (proper aggregation across weekly/monthly granularity) - Increase proxy timeout to 180s for slow BigQuery fetches - Reduce ci-metrics to 1 worker to avoid redundant cache warmups
ludamad
force-pushed
the
ad/fix/ci-metrics-deploy
branch
from
363fa18 to
77aff10
Compare
ludamad
changed the title
…ange - CloudTrail resolver now joins RunInstances + CreateTags events by instance ID, then matches to ci_runs via Dashboard and Name tags instead of bare timestamp proximity - Restore merge_train_failure_slack_notify to match base branch
The previous CloudTrail resolver had three issues causing near-zero match rates: 1. Single-pass event fetching hit the 5000-event pagination limit, missing most RunInstances events beyond ~16 days. Now fetches in daily chunks. 2. CreateTags filter discarded Name-only events (line 126 of aws_request_instance_type), losing the Name tag for ~90% of instances. Now accumulates all tags first, then filters by Group=build-instance. 3. Name tag parsing couldn't handle INSTANCE_POSTFIX suffixes (e.g. pr-123_arm64_a1-fast). Now uses regex to extract branch name regardless of postfix format. 4. Matching window was 10 minutes (only matched first CI step). Now allows 90 minutes to match all steps on an instance. Tested against real data: resolves 4187/4638 (90%) unknown instance types across 90 days of CloudTrail history.
The API was reading CI runs from a Redis+SQLite hybrid, but the hourly Redis sync used INSERT OR REPLACE which overwrote CloudTrail-enriched instance_type and cost_usd back to empty values. Now: - get_ci_runs() reads exclusively from SQLite - sync_ci_runs_to_sqlite() uses ON CONFLICT DO UPDATE that preserves enriched fields (only overwrites if Redis has non-empty values) - app.py calls updated to drop unused Redis connection argument
- Add hardcoded rates for m6a.xlarge/4xlarge/8xlarge/24xlarge that were missing, causing 192-vCPU fallback ($100+ instead of ~$8 for 8xlarge) - Make pricing discovery dynamic: query DB for distinct instance types so newly resolved types get live pricing automatically - Add recalculate_all_costs() to fix historical cost data
Instead of guessing 192 vCPUs (which massively overestimates), return None so the cost shows as unknown rather than a fabricated number.
ludamad
force-pushed
the
ad/fix/ci-metrics-deploy
branch
from
9a6cabb to
0245302
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
test_daily_statstable tracks daily passed/failed/flaked counts per test command without persisting individual passed events. Backfills from existingtest_eventson startup.log_ci_runnow prefersEC2_INSTANCE_TYPEenv var over metadata endpoint (which fails inside Docker containers). Fixes 99% of CI runs showing "unknown" instance type.RunInstancesevents and matching by timestamp proximity. Recalculates costs with correct instance rates.periodmetadata; dashboard headers show the active date range.Test plan
ci3/dashboard/deploy.shtest_daily_statstable created and backfilledsudo journalctl -u rkapp | grep cloudtrail