chore(test): pin fixture deps and manifest-drive the e2e harness by wyattjoh · Pull Request #280 · clerk/cli

wyattjoh · 2026-05-12T20:03:06Z

Summary

Reproducibility, harness ergonomics, and a pre-commit-hook fix for the e2e suite.

The e2e fixture tooling moves from bunx/bun install onto npx/npm ci with a committed package-lock.json in each fixture, and every non-Clerk dependency is resolved to an exact version through npm view at refresh time so scaffolded projects no longer drift between runs. Scaffolders that change behavior across majors are pinned through an explicit pinnedDependencyRanges config, replacing the older pinned: boolean + --force flag, and a TanStack peer-dependency mismatch is now expressed declaratively via packageJsonOverrides. The resolution, override, and validation logic lives in a new scripts/lib/fixture-deps.ts module covered by its own unit tests.

The e2e harness consolidates around a single createFixtureHarness(name) entry point. Every fixture's config moves into test/e2e/fixtures.manifest.ts keyed by fixture name, so the refresh script can import the manifest directly and no longer needs to load test files or the CLERK_REFRESH_FIXTURES env-var shim. The harness handle carries the fixture config and directory, exposes a per-test users API that auto-cleans created users in afterEach, and gives runFixtureTests/runFileExistsTest/runBrowserTests a single place to read config from. A dedicated test/e2e/tsconfig.json brings the e2e tree under bun run typecheck, and e2e logging is silenced unless CLERK_E2E_DEBUG=1.

The custom scripts/run-tests.ts wrapper is dropped in favor of running bun test --parallel directly, which implies --isolate and gives module-state isolation between test files for free. Lint and format coverage extend over test/e2e/, with an .oxfmtrc.json that excludes fixture lockfiles and a no-console oxlint override scoped to the e2e tree where the logger is a deliberate console wrapper.

Test plan

changeset-bot · 2026-05-12T20:03:10Z

🦋 Changeset detected

Latest commit: d4a7d82

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 0 packages

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

wyattjoh · 2026-05-12T20:03:10Z

Stack: wyattjoh/nextjs-pinned-range

chore(test): pin fixture deps and manifest-drive the e2e harness #280 👈 this PR
- refactor(test): standardize log capture on useCaptureLog #287

Part of a stacked-prs chain. Do not merge manually.

coderabbitai · 2026-05-12T20:05:35Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds a fixture-dependency utility module (scripts/lib/fixture-deps.ts) and tests, and integrates it into the refresh-e2e-fixtures script to apply package.json overrides, resolve dependency specs to exact versions, and validate pinned ranges. Migrates test log-capture from an AsyncLocalStorage-based API to a hook-style useCaptureLog() and replaces with a module-level active capture plus new log.ui() method; updates many tests, test stubs, and integration harnesses to use the new API. Updates spinner/logging to emit via log.ui(), refactors e2e fixture setup to createGetFixture()/Bun describe blocks and npx scaffolding, and pins numerous fixture package.json versions. Removes scripts/run-tests.ts.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e/users-list.test.ts`:
- Around line 48-52: The test switched to awaiting Promise.all for
createTestUser but never populates createdIds, so afterAll cleanup doesn't
delete the created users; modify the code that calls createTestUser (the
Promise.all(...) result assigned to users) to extract and push each returned
user id into the existing createdIds array (or replace createdIds with
users.map(u => u.id)), ensuring the afterAll teardown logic that iterates
createdIds will actually delete the created users; reference the createTestUser
call site and the createdIds variable and the afterAll cleanup to make the
change.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: d63bd112-853f-4241-b00b-b8d1bcbd7f7a

📥 Commits

Reviewing files that changed from the base of the PR and between 3e50dc0 and 1514c5e.

⛔ Files ignored due to path filters (9)

test/e2e/fixtures/astro/package-lock.json is excluded by !**/package-lock.json
test/e2e/fixtures/nextjs-app-router-next14/package-lock.json is excluded by !**/package-lock.json
test/e2e/fixtures/nextjs-app-router/package-lock.json is excluded by !**/package-lock.json
test/e2e/fixtures/nextjs-pages-router/package-lock.json is excluded by !**/package-lock.json
test/e2e/fixtures/nuxt/package-lock.json is excluded by !**/package-lock.json
test/e2e/fixtures/react-router/package-lock.json is excluded by !**/package-lock.json
test/e2e/fixtures/react/package-lock.json is excluded by !**/package-lock.json
test/e2e/fixtures/tanstack-start/package-lock.json is excluded by !**/package-lock.json
test/e2e/fixtures/vue/package-lock.json is excluded by !**/package-lock.json

📒 Files selected for processing (86)

.claude/rules/e2e.md
.claude/rules/logging.md
.oxfmtrc.json
.oxlintrc.json
CONTRIBUTING.md
package.json
packages/cli-core/package.json
packages/cli-core/src/cli-program.ts
packages/cli-core/src/commands/api/catalog.test.ts
packages/cli-core/src/commands/api/index.test.ts
packages/cli-core/src/commands/api/interactive.test.ts
packages/cli-core/src/commands/api/ls.test.ts
packages/cli-core/src/commands/apps/create.test.ts
packages/cli-core/src/commands/apps/list.test.ts
packages/cli-core/src/commands/auth/login.test.ts
packages/cli-core/src/commands/auth/logout.test.ts
packages/cli-core/src/commands/billing/index.test.ts
packages/cli-core/src/commands/config/pull.test.ts
packages/cli-core/src/commands/config/push.test.ts
packages/cli-core/src/commands/config/schema.test.ts
packages/cli-core/src/commands/deploy/index.test.ts
packages/cli-core/src/commands/doctor/context.test.ts
packages/cli-core/src/commands/env/pull.test.ts
packages/cli-core/src/commands/init/index.test.ts
packages/cli-core/src/commands/init/scan.test.ts
packages/cli-core/src/commands/link/index.test.ts
packages/cli-core/src/commands/open/index.test.ts
packages/cli-core/src/commands/orgs/index.test.ts
packages/cli-core/src/commands/switch-env/index.test.ts
packages/cli-core/src/commands/unlink/index.test.ts
packages/cli-core/src/commands/users/create.test.ts
packages/cli-core/src/commands/users/list.test.ts
packages/cli-core/src/commands/users/menu.test.ts
packages/cli-core/src/commands/users/open.test.ts
packages/cli-core/src/commands/whoami/index.test.ts
packages/cli-core/src/lib/auth-server.test.ts
packages/cli-core/src/lib/autoclaim.test.ts
packages/cli-core/src/lib/autolink.test.ts
packages/cli-core/src/lib/bapi-command.test.ts
packages/cli-core/src/lib/first-application.test.ts
packages/cli-core/src/lib/keyless.test.ts
packages/cli-core/src/lib/log.test.ts
packages/cli-core/src/lib/log.ts
packages/cli-core/src/lib/spinner.ts
packages/cli-core/src/test/integration/lib/harness.ts
packages/cli-core/src/test/lib/stubs.ts
scripts/lib/fixture-deps.test.ts
scripts/lib/fixture-deps.ts
scripts/refresh-e2e-fixtures.ts
scripts/run-tests.ts
test/e2e/astro.test.ts
test/e2e/fixtures/astro/README.md
test/e2e/fixtures/astro/package.json
test/e2e/fixtures/nextjs-app-router-next14/package.json
test/e2e/fixtures/nextjs-app-router/package.json
test/e2e/fixtures/nextjs-pages-router/package.json
test/e2e/fixtures/nuxt/package.json
test/e2e/fixtures/react-router/app/routes/home.tsx
test/e2e/fixtures/react-router/package.json
test/e2e/fixtures/react-router/tsconfig.json
test/e2e/fixtures/react-router/vite.config.ts
test/e2e/fixtures/react/index.html
test/e2e/fixtures/react/package.json
test/e2e/fixtures/tanstack-start/AGENTS.md
test/e2e/fixtures/tanstack-start/README.md
test/e2e/fixtures/tanstack-start/package.json
test/e2e/fixtures/vue/index.html
test/e2e/fixtures/vue/package.json
test/e2e/fixtures/vue/src/App.vue
test/e2e/fixtures/vue/src/components/HelloWorld.vue
test/e2e/lib/dev-server.ts
test/e2e/lib/fixture-setup.ts
test/e2e/lib/fixture-test.ts
test/e2e/lib/logger.ts
test/e2e/lib/test-user.ts
test/e2e/lib/types.ts
test/e2e/nextjs-app-router-next14.test.ts
test/e2e/nextjs-app-router.test.ts
test/e2e/nextjs-pages-router.test.ts
test/e2e/nuxt.test.ts
test/e2e/react-router.test.ts
test/e2e/react.test.ts
test/e2e/tanstack-start.test.ts
test/e2e/tsconfig.json
test/e2e/users-list.test.ts
test/e2e/vue.test.ts

💤 Files with no reviewable changes (4)

test/e2e/fixtures/react-router/app/routes/home.tsx
scripts/run-tests.ts
test/e2e/fixtures/react-router/tsconfig.json
test/e2e/fixtures/tanstack-start/AGENTS.md

✅ Files skipped from review due to trivial changes (18)

.oxfmtrc.json
test/e2e/fixtures/nuxt/package.json
test/e2e/fixtures/nextjs-pages-router/package.json
test/e2e/fixtures/astro/package.json
CONTRIBUTING.md
test/e2e/fixtures/vue/src/components/HelloWorld.vue
packages/cli-core/src/commands/users/menu.test.ts
packages/cli-core/src/commands/doctor/context.test.ts
test/e2e/fixtures/vue/src/App.vue
test/e2e/fixtures/vue/index.html
test/e2e/fixtures/tanstack-start/README.md
packages/cli-core/src/cli-program.ts
packages/cli-core/package.json
test/e2e/fixtures/react/index.html
test/e2e/fixtures/nextjs-app-router-next14/package.json
.oxlintrc.json
packages/cli-core/src/lib/auth-server.test.ts
test/e2e/fixtures/astro/README.md

🚧 Files skipped from review as they are similar to previous changes (7)

test/e2e/fixtures/nextjs-app-router/package.json
test/e2e/fixtures/react-router/package.json
scripts/lib/fixture-deps.test.ts
test/e2e/lib/types.ts
scripts/lib/fixture-deps.ts
scripts/refresh-e2e-fixtures.ts
test/e2e/fixtures/vue/package.json

…ture lifecycle Pin every E2E fixture project's dependencies and check in package-lock.json files so fixture installs are reproducible and skip scaffolding the upstream templates from scratch on each run. Switch fixtures from bun-based installs to npm to match what users would do on a freshly scaffolded app. Replace the useFixture helper with createGetFixture, returning { fixture, users } so each test owns its own user lifecycle and tracks created users for explicit cleanup. Add scripts/lib/fixture-deps for the shared pinning logic plus tests. Migrate test infrastructure off scripts/run-tests.ts to native bun test --parallel for both unit and E2E pipelines, and fold test/e2e into the lint, format, and typecheck commands.

Each fixture test file calls `describe()` at the top level, so the refresh script could no longer load them outside `bun test`. Move every fixture's config into `test/e2e/fixtures.manifest.ts` keyed by fixture name, and have `createFixtureHarness(name)` look up the config plus fixtureDir from the manifest, embedding both on the returned `Fixture`. The refresh script imports the manifest directly, dropping all test-file imports and the `CLERK_REFRESH_FIXTURES` env-var shim. Each fixture test file shrinks to a typed `createFixtureHarness("<name>")` call, and the helpers (`runFixtureTests`, `runFileExistsTest`, `runBrowserTests`) read config from the harness handle instead of threading it through every call.

`oxfmt`/`oxlint` exit with an error when every staged file is ignored by the formatter/linter config, which breaks commits that only touch `test/e2e/fixtures/**`. Pass `--no-error-on-unmatched-pattern` to both so fixture-only commits no longer fail the pre-commit hook.

Inline CLERK_PLATFORM_API_KEY plumbing per CLI invocation instead of a shared helper, validate publishable/secret keys with regex expectations, and force color in the op-wrapped subprocess for readable output.

Regenerates every fixture via `bun run e2e:refresh-fixtures` to pick up upstream template/dependency drift since the harness overhaul landed.

…ange lookup - scripts/lib/fixture-deps.ts: parallelize per-field resolveVersion calls via Promise.all - scripts/lib/fixture-deps.ts: replace nested ternary in validatePinnedDependencyRanges with Array.find

EFAULT from node:fs/promises rm in afterAll was failing the Nuxt fixture even though the test itself passed (oven-sh/bun#28958 and #9298 surface EFAULT in place of the real errno). Wrap fixture-setup cleanup in a best-effort safeRm that logs and continues; the OS reclaims /tmp anyway.

wyattjoh changed the title ~~wyattjoh/nextjs pinned range~~ chore(test): pin e2e fixture dependencies and migrate to npm May 12, 2026

wyattjoh force-pushed the wyattjoh/nextjs-pinned-range branch from 3e50dc0 to 1514c5e Compare May 13, 2026 02:19

coderabbitai Bot reviewed May 13, 2026

View reviewed changes

Comment thread test/e2e/users-list.test.ts Outdated

wyattjoh force-pushed the wyattjoh/nextjs-pinned-range branch 2 times, most recently from 88c14ab to 2ca4be0 Compare May 13, 2026 19:51

wyattjoh mentioned this pull request May 13, 2026

refactor(test): standardize log capture on useCaptureLog #287

Open

4 tasks

wyattjoh force-pushed the wyattjoh/nextjs-pinned-range branch from 2ca4be0 to 5e0198f Compare May 13, 2026 20:03

wyattjoh marked this pull request as draft May 13, 2026 20:28

wyattjoh marked this pull request as ready for review May 14, 2026 18:13

wyattjoh force-pushed the wyattjoh/nextjs-pinned-range branch from 5e0198f to 7750d4a Compare May 14, 2026 18:13

wyattjoh changed the title ~~chore(test): pin e2e fixture dependencies and migrate to npm~~ chore(test): pin fixture deps and manifest-drive the e2e harness May 14, 2026

wyattjoh added 6 commits May 14, 2026 12:42

docs(changeset): add empty changeset

c171bc3

test(e2e): tighten fixture setup env handling and key validation

3737cae

Inline CLERK_PLATFORM_API_KEY plumbing per CLI invocation instead of a shared helper, validate publishable/secret keys with regex expectations, and force color in the op-wrapped subprocess for readable output.

test(e2e): replace key regex with presence checks in fixture setup

8881d1d

wyattjoh force-pushed the wyattjoh/nextjs-pinned-range branch from 710f7b0 to 8881d1d Compare May 14, 2026 18:53

chore(test): refresh e2e fixtures

7a4c718

Regenerates every fixture via `bun run e2e:refresh-fixtures` to pick up upstream template/dependency drift since the harness overhaul landed.

rafa-thayto requested changes May 14, 2026

View reviewed changes

Comment thread scripts/lib/fixture-deps.ts Outdated

Comment thread scripts/lib/fixture-deps.ts

refactor(scripts): batch fixture dependency resolution and simplify r…

1444b27

…ange lookup - scripts/lib/fixture-deps.ts: parallelize per-field resolveVersion calls via Promise.all - scripts/lib/fixture-deps.ts: replace nested ternary in validatePinnedDependencyRanges with Array.find

wyattjoh requested a review from rafa-thayto May 14, 2026 19:52

rafa-thayto approved these changes May 14, 2026

View reviewed changes

wyattjoh merged commit 6ac3fa7 into main May 14, 2026
10 checks passed

wyattjoh deleted the wyattjoh/nextjs-pinned-range branch May 14, 2026 21:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(test): pin fixture deps and manifest-drive the e2e harness#280

chore(test): pin fixture deps and manifest-drive the e2e harness#280
wyattjoh merged 9 commits into
mainfrom
wyattjoh/nextjs-pinned-range

wyattjoh commented May 12, 2026 •

edited

Loading

Uh oh!

changeset-bot Bot commented May 12, 2026 •

edited

Loading

Uh oh!

wyattjoh commented May 12, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented May 12, 2026 •

edited

Loading

Reviews paused

Walkthrough

Estimated code review effort

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

wyattjoh commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

changeset-bot Bot commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

wyattjoh commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Estimated code review effort

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

wyattjoh commented May 12, 2026 •

edited

Loading

changeset-bot Bot commented May 12, 2026 •

edited

Loading

wyattjoh commented May 12, 2026 •

edited

Loading

coderabbitai Bot commented May 12, 2026 •

edited

Loading