Skip to content

Conversation

@keugenek
Copy link
Contributor

Summary

  • Add GitHub Actions workflow for Apps-MCP continuous evaluations
  • Workflow triggers nightly (2am UTC) and on version releases (v* tags)
  • Builds CLI binary from current branch and uploads to UC Volume
  • Runs generation and evaluation jobs from neondatabase/eng-app-devex bundle

Related

Test plan

  • Create apps-mcp-evals environment in repo settings
  • Add DATABRICKS_HOST and DATABRICKS_TOKEN secrets
  • Trigger workflow manually via workflow_dispatch
  • Verify jobs complete successfully
  • Check MLflow experiment at /Shared/apps-mcp-evaluations-staging

🤖 Generated with Claude Code

Add GitHub Actions workflow that:
- Triggers nightly (2am UTC) and on version releases
- Builds CLI binary for Linux
- Uploads binary to UC Volume
- Deploys and runs eval jobs from neondatabase/eng-app-devex

Bundle located at: apps-mcp-evals/ subdirectory

Requires 'apps-mcp-evals' environment with DATABRICKS_HOST and
DATABRICKS_TOKEN secrets.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@keugenek keugenek force-pushed the feature/apps-mcp-evals-cicd branch from add5c11 to acd2449 Compare January 16, 2026 15:38
@eng-dev-ecosystem-bot
Copy link
Collaborator

eng-dev-ecosystem-bot commented Jan 16, 2026

Commit: 8cd23f1

Run: 21135742811

Env ❌​FAIL 🟨​KNOWN 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
🟨​ aws linux 7 10 2 411 695 23:50
🟨​ azure windows 4 8 3 415 692 18:58
❌​ gcp windows 2 11 1 3 402 698 22:30
21 interesting tests: 17 KNOWN, 2 FAIL, 1 SKIP, 1 RECOVERED
Test Name aws linux azure windows gcp windows
🟨​ TestAccept 🟨​K 🟨​K 🟨​K
🟨​ TestAccept/bundle/deployment/bind/alert 🙈​S 💚​R 🟨​K
❌​ TestAccept/bundle/deployment/bind/alert/DATABRICKS_BUNDLE_ENGINE=direct ✅​p ❌​F
❌​ TestAccept/bundle/deployment/bind/alert/DATABRICKS_BUNDLE_ENGINE=terraform ✅​p ❌​F
🟨​ TestAccept/bundle/generate/alert 💚​R 🟨​K 🟨​K
🟨​ TestAccept/bundle/generate/alert/DATABRICKS_BUNDLE_ENGINE=direct 💚​R 🟨​K 🟨​K
🟨​ TestAccept/bundle/generate/alert/DATABRICKS_BUNDLE_ENGINE=terraform 💚​R 🟨​K 🟨​K
🟨​ TestAccept/bundle/resources/alerts/basic 💚​R 💚​R 🟨​K
🟨​ TestAccept/bundle/resources/alerts/basic/DATABRICKS_BUNDLE_ENGINE=direct 💚​R 💚​R 🟨​K
🟨​ TestAccept/bundle/resources/alerts/basic/DATABRICKS_BUNDLE_ENGINE=terraform 💚​R 💚​R 🟨​K
🟨​ TestAccept/bundle/resources/alerts/with_file 💚​R 💚​R 🟨​K
🟨​ TestAccept/bundle/resources/alerts/with_file/DATABRICKS_BUNDLE_ENGINE=direct 💚​R 💚​R 🟨​K
🟨​ TestAccept/bundle/resources/alerts/with_file/DATABRICKS_BUNDLE_ENGINE=terraform 💚​R 💚​R 🟨​K
🙈​ TestAccept/bundle/resources/permissions 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions 🟨​K 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions 🟨​K 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K
💚​ TestAccept/ssh/connection 💚​R 💚​R 💚​R
Top 10 slowest tests (at least 2 minutes):
duration env testname
6:08 aws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:32 aws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:30 gcp windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:14 gcp windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
3:31 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:12 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:50 aws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:42 aws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:17 azure windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:12 azure windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct

@keugenek
Copy link
Contributor Author

Move triggering CICD to apps-devx repo

@keugenek keugenek closed this Jan 19, 2026
@pietern pietern deleted the feature/apps-mcp-evals-cicd branch January 19, 2026 11:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants