From 903220a3870bb1561796975c3fbcb526ace87557 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 21 Mar 2026 00:24:31 +0000 Subject: [PATCH 1/2] Initial plan From 7d9f7751c4bbe79fe5a02e2d5d621f2add79ddb4 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 21 Mar 2026 00:33:30 +0000 Subject: [PATCH 2/2] docs: accept ADR-0021 monorepo decision with ecosystem and extraction analysis Co-authored-by: JustAGhosT <5531814+JustAGhosT@users.noreply.github.com> Agent-Logs-Url: https://github.com/JustAGhosT/codeflow-engine/sessions/8069b331-8ba2-46d3-88c9-3aa6120c0e20 --- MIGRATION_PLAN.md | 59 ++-- docs/adr/0021-repository-structure.md | 404 +++++++++++--------------- docs/adr/README.md | 9 +- 3 files changed, 212 insertions(+), 260 deletions(-) diff --git a/MIGRATION_PLAN.md b/MIGRATION_PLAN.md index bc08172..ebc99ff 100644 --- a/MIGRATION_PLAN.md +++ b/MIGRATION_PLAN.md @@ -1,45 +1,62 @@ # CodeFlow Monorepo Migration Plan -This document tracks the migration of all CodeFlow-related repositories into a unified monorepo structure. +This document tracks the migration of all CodeFlow-related repositories into a unified +monorepo structure. The decision to consolidate is recorded in +[docs/adr/0021-repository-structure.md](docs/adr/0021-repository-structure.md). ## Structure -- engine/ # Python core engine project -- desktop/ # Electron/Tauri app -- vscode-extension/ # VS Code extension -- website/ # Docs/marketing -- orchestration/ # Infra, bootstrap, and shared orchestration assets -- templates/ # Workflow templates -- tools/ # Dev tools/scripts -- tests/ # Unified test suite for engine -- docs/ # Shared documentation +``` +engine/ # Python core engine project +desktop/ # Tauri + React desktop application +vscode-extension/ # VS Code extension +website/ # Next.js marketing and documentation site +orchestration/ # Azure IaC, bootstrap scripts, and shared utility packages +docs/ # Shared project documentation +tools/ # Shared development tooling and helper scripts +``` ## Current Status -Completed: +### Phase 1 — Code Consolidation ✅ Complete 1. Added monorepo migration scaffolding and shared documentation. 2. Imported `codeflow-desktop` into `desktop/`. 3. Imported `codeflow-website` into `website/`. 4. Imported `codeflow-orchestration` into `orchestration/`. 5. Imported `codeflow-vscode-extension` into `vscode-extension/`. +6. Added path-aware CI workflow (`.github/workflows/monorepo-ci.yml`). +7. Added archive and redirect guidance (`docs/LEGACY_REPO_REDIRECTS.md`). +8. Finalised repository structure decision (ADR-0021 accepted). -Pending: +### Phase 2 — Tooling Standardisation ⏳ In Progress -1. Normalize dependency management across Python and Node-based projects. -2. Add path-aware CI and release automation. -3. Consolidate duplicate docs, licenses, and contribution guidance. -4. Complete archive and redirect steps for the former split repositories. +1. Normalise dependency management across Python and Node.js projects. +2. Add path-aware release automation for each component. +3. Consolidate duplicate `README`, `LICENSE`, and `CONTRIBUTING` files. +4. Archive legacy split repositories and update their READMEs to redirect here. -## Migration Steps +## Future Extraction Candidates -1. Import code from each repo, preserving git history. +The following components *could* be extracted to separate repositories in future if +specific conditions are met (see ADR-0021 for full rationale): + +| Component | Condition for extraction | +|---|---| +| `website/` | Content team needs write access without engine write access | +| `orchestration/bootstrap/` | A shared `justaghost/*` or `phoenixvc/*` infra repo is created | + +No extraction is recommended at this stage. + +## Migration Steps (Reference) + +1. Import code from each repo, preserving git history (`git subtree`). 2. Resolve conflicts and update imports. -3. Centralize CI/CD workflows. +3. Centralise CI/CD workflows with path-aware filtering. 4. Update documentation and onboarding. -5. Deprecate old repositories. +5. Archive legacy split repositories. 6. Announce migration and monitor feedback. --- -For detailed instructions, see MIGRATION_GUIDE.md. +For detailed instructions, see [MIGRATION_GUIDE.md](MIGRATION_GUIDE.md). diff --git a/docs/adr/0021-repository-structure.md b/docs/adr/0021-repository-structure.md index 493eb16..c47b59a 100644 --- a/docs/adr/0021-repository-structure.md +++ b/docs/adr/0021-repository-structure.md @@ -1,303 +1,237 @@ -# 21. Repository Structure and Monorepo vs Multi-Repo Strategy +# 21. Repository Structure: Monorepo vs Multi-Repo Decision ## Status -Proposed +Accepted ## Date 2025-12-08 +## Updated + +2026-03-21 + ## Context -The codeflow-engine project has grown significantly and now encompasses multiple distinct concerns: +The CodeFlow project grew to span multiple distinct concerns across several archived +`codeflow-*` repositories under the `JustAGhosT` GitHub organization: -1. **Core Engine**: The main automation and workflow engine -2. **Web UI/Dashboard**: React-based user interface -3. **VS Code Extension**: Editor integration -4. **Desktop Application**: Electron-based desktop app (codeflow-desktop) -5. **Documentation Website**: Marketing and documentation site -6. **Template System**: Reusable workflow templates -7. **Infrastructure**: Kubernetes, Terraform, deployment configs +| Repository | Status before migration | +|---|---| +| `codeflow-engine` | Active (this repo) | +| `codeflow-orchestration` | Archived | +| `codeflow-infrastructure` | Archived | +| `codeflow-desktop` | Archived | +| `codeflow-azure-setup` | Archived | +| `codeflow-website` | Archived | +| `codeflow-vscode-extension` | Archived | -Currently, all these components live in a single monorepo, which has both advantages and growing pains. +The discussion ([GitHub Issue #21](https://github.com/JustAGhosT/codeflow-engine/issues/21)) +asked two related questions: -### Current Repository Structure +1. Should the `codeflow-*` repositories be reintegrated into a monorepo? +2. How do the `codeflow-*` repositories fit into the broader `phoenixvc/*` and + `justaghost/*` ecosystem? -``` -codeflow-engine/ -├── engine/ # Python package workspace -│ └── codeflow_engine/ # Canonical Python package -├── desktop/ # Electron desktop app -├── vscode-extension/ # VS Code extension -├── website/ # Documentation/marketing site -├── templates/ # Workflow templates -├── docs/ # Documentation -├── tools/ # Development tools -└── tests/ # Test suite -``` +This ADR records the decision made and the reasoning behind it. -### Problems with Current Structure +--- -1. **Build Complexity**: Single CI/CD pipeline for multiple technologies -2. **Dependency Conflicts**: Python, Node.js, .NET tools in one repo -3. **Release Cycles**: Different components need different release schedules -4. **Team Boundaries**: Different teams work on different components -5. **Clone Size**: Large repository slows initial clones and fetches -6. **Testing**: Full test suite runs even for unrelated changes -7. **Permissions**: Difficult to grant granular access to components +## Component Summary -## Decision Options +Before deciding on structure, the components were analysed for purpose and coupling: -### Option 1: Keep Monorepo with Better Organization +| Component | Type | Direct code deps | +|---|---|---| +| `engine/` | Python library + service | None (only external APIs) | +| `desktop/` | Tauri + React app | `engine/` via HTTP/WebSocket sidecar only | +| `vscode-extension/` | VS Code extension (TypeScript) | `engine/` via HTTP REST only | +| `website/` | Next.js marketing/docs site | None (fully standalone) | +| `orchestration/` | Azure IaC + shared utility packages | None | -Maintain single repository but improve tooling and structure. +Key observation: `desktop/` and `vscode-extension/` communicate with `engine/` through +well-defined API contracts, not direct imports. They are **loosely coupled at the code +level** and can be built, released, and deployed independently. -**Pros:** -- Easy code sharing and refactoring across components -- Single source of truth for all code -- Simplified dependency version coordination -- Easy to enforce consistent coding standards -- Better for atomic cross-component changes +--- -**Cons:** -- CI/CD complexity continues to grow -- Large repository size -- Difficult selective access control -- All-or-nothing cloning - -**Tooling:** -- Use Nx, Turborepo, or Bazel for monorepo management -- Implement affected testing (only test changed components) -- Use sparse checkout for developers - -### Option 2: Split into 3-4 Core Repositories - -Separate concerns into logical repositories while maintaining some cohesion. - -**Proposed Split:** - -1. **codeflow-engine** (Core Python Package) - - Python engine code - - API server - - Worker processes - - Core tests - - Python dependencies only - -2. **codeflow-ui** (Frontend Applications) - - Web dashboard (React) - - VS Code extension - - Desktop app (Electron) - - Shared UI components - - Frontend tests - -3. **codeflow-infrastructure** (Deployment & Operations) - - Kubernetes manifests - - Terraform/Bicep configs - - Docker configurations - - Monitoring setups - - Infrastructure as Code - -4. **codeflow-templates** (Template Library) - - Workflow templates - - Platform-specific configs - - Template documentation - - Template tests +## Options Evaluated + +### Option 1 — Keep monorepo with path-aware CI ✅ (Chosen) + +Consolidate all `codeflow-*` repositories into this repo; use path filters in CI so +only the affected component builds on each change. **Pros:** -- Clear separation of concerns -- Independent release cycles -- Focused CI/CD pipelines -- Smaller, faster clones -- Better team boundaries -- Granular access control +- Archived repos need no ongoing maintenance in isolation. +- Atomic cross-component changes remain easy (e.g. API shape change + client update in + one PR). +- Single source of truth for linting configs, licences, and contribution guides. +- Path-aware CI (`dorny/paths-filter`) eliminates the full-build penalty. +- Small team; cross-repo coordination overhead is not justified. **Cons:** -- Cross-repo coordination needed -- Duplicate some tooling/configs -- More complex version management -- Breaking changes harder to coordinate -- Need robust versioning strategy - -### Option 3: Full Microrepo (Many Small Repos) - -Split into many small, focused repositories. - -**Proposed Structure:** -- codeflow-engine-core -- codeflow-engine-api -- codeflow-engine-worker -- codeflow-ui-web -- codeflow-ui-vscode -- codeflow-desktop -- codeflow-infra-k8s -- codeflow-infra-terraform -- codeflow-templates +- Larger clone size (mitigated by sparse checkout or shallow clone). +- Granular per-component access control is not possible (not a current requirement). -**Pros:** -- Maximum independence -- Clearest boundaries -- Finest-grained access control -- Smallest repositories +### Option 2 — Split into 3–4 focused repositories + +Extract engine, UI clients, infrastructure, and templates into separate repos. + +**Pros:** Independent release cycles, focused pipelines, smaller clones. **Cons:** -- Coordination nightmare -- Versioning complexity -- Duplicated tooling everywhere -- Difficult cross-component refactoring -- High overhead for atomic changes +- The component repos were already archived; re-splitting re-creates maintenance burden. +- Cross-repo version coordination is non-trivial for a small team. +- No immediate access-control requirement justifies the overhead. -## Recommendation +### Option 3 — Full microrepo (one repo per component) -**Choose Option 2**: Split into 3-4 core repositories for optimal balance between independence and coordination. +Maximum independence but maximum coordination overhead. Not appropriate for the team +size. -### Rationale +--- -1. **Clear Boundaries**: The proposed split aligns with natural team and technology boundaries -2. **Manageable Complexity**: 3-4 repos is manageable without excessive coordination overhead -3. **Independent Deployments**: Core engine, UI, and infrastructure can deploy independently -4. **Reasonable Clone Times**: Each repo will be significantly smaller -5. **Focused CI/CD**: Each repo can have optimized pipelines for its technology -6. **Gradual Migration**: Can split incrementally, starting with highest-value separations +## Decision -## Implementation Plan +**Consolidate all `codeflow-*` repositories into this monorepo (Option 1).** -### Phase 1: Preparation (Weeks 1-2) +All formerly separate repositories have been imported with git history preserved: -1. **Audit Dependencies**: Map all inter-component dependencies -2. **Design Interfaces**: Define stable APIs between components -3. **Version Strategy**: Establish versioning scheme (semantic versioning + matrix) -4. **CI/CD Planning**: Design independent CI/CD pipelines -5. **Migration Scripts**: Prepare git history preservation scripts +| Legacy repository | Monorepo path | +|---|---| +| `codeflow-engine` | `engine/` | +| `codeflow-desktop` | `desktop/` | +| `codeflow-website` | `website/` | +| `codeflow-orchestration` | `orchestration/` | +| `codeflow-vscode-extension` | `vscode-extension/` | -### Phase 2: Core Engine Split (Weeks 3-4) +The canonical layout is: -1. Extract `engine/codeflow_engine/` to new `codeflow-engine` repository -2. Keep git history intact (use `git filter-repo`) -3. Set up independent CI/CD -4. Publish to PyPI from new repo -5. Update documentation +``` +codeflow-engine/ +├── engine/ # Python core package (codeflow_engine) +├── desktop/ # Tauri + React desktop application +├── vscode-extension/ # VS Code extension +├── website/ # Next.js marketing and documentation site +├── orchestration/ # Azure IaC, bootstrap scripts, shared utility packages +├── docs/ # Shared project documentation and ADRs +└── tools/ # Shared development tooling and helper scripts +``` -### Phase 3: UI Split (Weeks 5-6) +--- -1. Extract frontend code to `codeflow-ui` repository -2. Consolidate UI components and shared libraries -3. Set up Node.js/TypeScript CI/CD -4. Establish npm publishing workflow -5. Update integration points +## Was the Monorepo the Right Decision? -### Phase 4: Infrastructure Split (Weeks 7-8) +Short answer: **yes, for the current stage of the project**. -1. Extract infrastructure code to `codeflow-infrastructure` repository -2. Set up GitOps workflows -3. Establish environment management -4. Document deployment processes -5. Update runbooks +### Rationale -### Phase 5: Templates Split (Weeks 9-10) +1. **Most component repos were already archived.** Re-splitting means creating and + maintaining new repos for code that was intentionally deprioritised. +2. **Loose API coupling means monorepo risk is low.** Because `desktop/` and + `vscode-extension/` integrate with `engine/` through HTTP/WebSocket APIs and not + direct imports, independent release cadences are preserved even inside the same repo. +3. **Path-aware CI eliminates the main monorepo penalty.** The + `.github/workflows/monorepo-ci.yml` workflow uses `dorny/paths-filter` so only + the changed component's jobs run. +4. **Small team; cross-repo overhead is a real cost.** Version matrix management, + cross-repo PR coordination, and duplicated tooling configs are disproportionately + costly. +5. **History is preserved.** `git subtree` imports keep blame and history intact + without requiring any git tricks at development time. -1. Extract templates to `codeflow-templates` repository -2. Set up template validation pipeline -3. Establish template versioning -4. Create template marketplace -5. Document contribution guidelines +### What Could Reasonably Be Extracted Later -### Phase 6: Stabilization (Weeks 11-12) +The decision is **correct now** but two components warrant review as the project scales: -1. Cross-repo integration testing -2. Documentation updates -3. Developer onboarding guides -4. Automated cross-repo tools -5. Monitoring and alerting setup +#### `website/` — borderline case -## Inter-Repository Coordination +- The marketing/docs site has **zero code dependencies** on the engine or any other + component. It is a fully standalone Next.js application. +- It could be extracted to a dedicated repo (e.g. `codeflow-website`) if: + - A content team without engine write access needs to contribute, or + - The deployment cadence diverges significantly from the engine. +- **Recommendation:** Keep in the monorepo for now. If contributor access-control + needs arise, extract at that point. -### Version Matrix +#### `orchestration/bootstrap/` — generic Azure tooling -Maintain compatibility matrix across repositories: +- The PowerShell scripts in `orchestration/bootstrap/` are deliberately generic (they + create Azure resource groups, storage accounts, and Log Analytics workspaces for + any project). +- If a shared `justaghost/*` or `phoenixvc/*` infrastructure repo is established, + these scripts are good candidates to contribute upstream. +- **Recommendation:** Keep here until a target shared repo exists. Do not extract + speculatively. -```yaml -# codeflow-engine v1.2.0 compatibility -compatible_ui_versions: "^2.1.0" -compatible_templates: "^1.0.0" -minimum_infra_version: "1.3.0" -``` +--- -### Shared Tooling +## Ecosystem Fit: `phoenixvc/*` and `justaghost/*` -Maintain shared configurations in a `codeflow-shared` repository: -- Linting configs (ESLint, Ruff, Prettier) -- GitHub Actions workflows (reusable) -- Development environment configs -- Code generation templates +As of this decision there are **no code-level references** to `phoenixvc` anywhere in +this repository. The CodeFlow components relate to the broader organisation ecosystem +as follows: -### Communication +| Component | Role in the ecosystem | +|---|---| +| `engine/` | Authoritative backend for AI-powered PR automation; exposed as both a PyPI package and a self-hosted service. Any `justaghost/*` or `phoenixvc/*` project can consume it as a dependency. | +| `desktop/` | Local developer tooling for managing CodeFlow without a browser. Targets individual developers and small teams; no org-specific coupling. | +| `vscode-extension/` | IDE integration surface; publishes to the VS Code Marketplace. Usable by any developer regardless of org. | +| `website/` | Public-facing marketing and documentation. Not org-specific in content. | +| `orchestration/` | Azure IaC and bootstrap tooling for CodeFlow deployments. The generic bootstrap scripts could serve as a template for other `justaghost/*` or `phoenixvc/*` projects that run on Azure. | -1. **Cross-Repo Issues**: Use GitHub Projects to track cross-repo work -2. **Breaking Changes**: RFC process for changes affecting multiple repos -3. **Release Coordination**: Coordinated release schedule published quarterly -4. **Dependencies**: Renovate bot for automated dependency updates +### Integration path if a shared org repo is created -## Success Metrics +If a `justaghost/shared-infra` or `phoenixvc/platform-bootstrap` repository is +created in future, the recommended migration is: -- Build time reduction: Target 50% faster builds per repo -- Clone time: Under 2 minutes for any single repo -- Test execution: Under 10 minutes for affected tests -- Release frequency: Each component can release independently -- Developer onboarding: New developers can work on single component without full codebase +1. Extract `orchestration/bootstrap/` scripts to the shared repo. +2. Extract the generic utility packages (`@codeflow/utils`, + `codeflow-utils-python`) if other org projects will reuse them. +3. Keep CodeFlow-specific IaC (`orchestration/infrastructure/`) here. -## Consequences +No extraction is warranted until a concrete target repo exists. -### Positive +--- -- **Faster Development**: Focused repos mean faster builds and tests -- **Clear Ownership**: Each repo has clear team ownership -- **Independent Releases**: Components can evolve at their own pace -- **Better Security**: Granular access control per component -- **Reduced Conflicts**: Fewer merge conflicts with smaller teams per repo -- **Optimized Tooling**: Each repo can use best tools for its tech stack +## Implementation Status -### Negative +### Completed -- **Coordination Overhead**: Cross-repo changes require more planning -- **Duplicated Config**: Some configs will be duplicated across repos -- **Version Management**: Need to track compatible versions across repos -- **Onboarding Complexity**: New developers need to understand multi-repo structure -- **Migration Effort**: Significant work to split and test the separation +- All `codeflow-*` repositories imported with git history preserved. +- Path-aware monorepo CI workflow (`.github/workflows/monorepo-ci.yml`). +- Shared documentation under `docs/`. +- Archive and redirect guidance in `docs/LEGACY_REPO_REDIRECTS.md`. +- Migration documentation in `MIGRATION_PLAN.md` and `MIGRATION_GUIDE.md`. -### Mitigations +### Remaining Work -- Use monorepo tools temporarily to ease transition -- Establish clear inter-repo contracts and APIs -- Automate cross-repo testing with integration test suite -- Create comprehensive migration and onboarding documentation -- Use shared configuration repository for common tooling +1. Normalise dependency management across Python and Node.js components. +2. Add path-aware release automation for each component. +3. Consolidate duplicate `README`, `LICENSE`, and `CONTRIBUTING` files. +4. Archive legacy split repositories and update their READMEs to redirect here. -## Alternative Considered +--- -### Hybrid: Monorepo with Workspace Isolation +## Consequences -Use tools like Nx or Turborepo to get multi-repo benefits while keeping monorepo: +### Positive -``` -codeflow-engine/ -├── packages/ -│ ├── core/ # Python package -│ ├── web/ # React app -│ ├── vscode/ # VS Code extension -│ └── desktop/ # Electron app -├── nx.json -└── workspace.json -``` +- Single clone gives a contributor everything they need. +- Atomic cross-component changes require only one PR. +- Linting, formatting, and CI standards are enforced centrally. +- No ongoing maintenance of multiple archived repositories. -**Decision**: Not chosen because it doesn't solve clone size, access control, or technology heterogeneity issues. +### Negative -## Timeline +- Repository clone size is larger than any individual component (mitigated by sparse + checkout or shallow clone). +- Granular per-component access control is not possible inside GitHub's permission + model (not a current requirement). -- **Q1 2026**: Complete evaluation and decision -- **Q2 2026**: Execute split (Phases 1-6) -- **Q3 2026**: Stabilization and optimization -- **Q4 2026**: Evaluate and iterate +--- ## Related Decisions @@ -308,7 +242,7 @@ codeflow-engine/ ## References -- Monorepo vs Multi-Repo: https://monorepo.tools/ -- Google's Monorepo Experience: https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/ -- Git Filter-Repo: https://github.com/newren/git-filter-repo -- Nx Monorepo Tools: https://nx.dev/ +- Monorepo vs Multi-Repo: +- Google's Monorepo Experience: +- Migration Plan: [MIGRATION_PLAN.md](../../MIGRATION_PLAN.md) +- Legacy Redirects: [docs/LEGACY_REPO_REDIRECTS.md](../LEGACY_REPO_REDIRECTS.md) diff --git a/docs/adr/README.md b/docs/adr/README.md index 33a7ddd..d330935 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -32,9 +32,10 @@ Each ADR follows this structure: - **Supersedes ADR-0001**: Documents the actual Python-only implementation - Includes migration summary explaining why hybrid approach wasn't implemented -- [ADR-0021: Repository Structure and Monorepo vs Multi-Repo](0021-repository-structure.md) - **Status: Proposed** - - Evaluates splitting the monorepo into 3-4 focused repositories - - Includes detailed implementation plan and timeline +- [ADR-0021: Repository Structure and Monorepo vs Multi-Repo](0021-repository-structure.md) - **Status: Accepted** ⭐ + - **Decision**: Consolidate all `codeflow-*` repositories into this monorepo with path-aware CI + - Analyses ecosystem fit within `justaghost/*` and `phoenixvc/*` organisations + - Identifies `website/` and `orchestration/bootstrap/` as candidates for future extraction ### Communication & Integration @@ -145,7 +146,7 @@ The project uses a **Python-only architecture** (ADR-0019): ### Future Considerations -- **Repository Split** (ADR-0021): Evaluating split into 3-4 focused repos for better scalability +- **Repository Structure** (ADR-0021, Accepted): Monorepo consolidation complete. `website/` and `orchestration/bootstrap/` are candidates for future extraction if org-wide shared repos are created. - **Performance Optimization**: If needed, selective migration of hot paths to Rust/Go - **Microservices**: Potential extraction of high-throughput services