From f0fdbf564c38a6d418c4d5fd04f9f64275e11c8d Mon Sep 17 00:00:00 2001 From: Val Redchenko Date: Tue, 21 Apr 2026 16:43:03 +0100 Subject: [PATCH] docs: fix incorrect claims, add missing tools, and release guide Fix factual errors identified in docs audit (#187): - Correct ARIA definition (was claimed to be image analysis system) - Replace phantom tools in api-server.md with actual simulator - Rewrite generate-docs.md (was referencing nonexistent tox command) - Fix database.md migration list (enumerate all 6, replace stale ID) - Document 4 missing tools: check_schema_drift, db_table_totals, generate_api_docs, makeiso - Add release procedure guide covering all 3 releasable packages - Link releasing and PyPI setup docs in operations index --- docs/backend/api-server.md | 8 +- docs/backend/database.md | 8 +- docs/development/generate-docs.md | 44 ++++++- docs/development/tools.md | 51 ++++++++ docs/glossary.md | 6 +- docs/operations/index.md | 7 ++ docs/operations/releasing.md | 198 ++++++++++++++++++++++++++++++ 7 files changed, 308 insertions(+), 14 deletions(-) create mode 100644 docs/operations/releasing.md diff --git a/docs/backend/api-server.md b/docs/backend/api-server.md index d88f726..6a6b566 100644 --- a/docs/backend/api-server.md +++ b/docs/backend/api-server.md @@ -5,7 +5,7 @@ The core backend service providing HTTP API, database operations, and message qu ## Backend Operations ```bash -# create env and launch service stack locally: +# create env and launch service stack locally (from smartem-devtools checkout): ./scripts/k8s/dev-k8s.sh up # launch RabbitMQ worker (consumer) @@ -13,9 +13,9 @@ python -m smartem_backend.consumer # ERROR level (default) python -m smartem_backend.consumer -v # INFO level python -m smartem_backend.consumer -vv # DEBUG level -# simulating an system event: -python -m smartem_backend.simulate_msg --help # to see a list of options -./tools/simulate-messages.sh # run a simulation, triggering system events in sequence +# simulating a system event (see tools.md for full usage): +uv run python tools/external_message_simulator.py list-messages # see available message types +uv run python tools/external_message_simulator.py workflow-simulation --gridsquare-id "DEV_001" # run a workflow # run HTTP API (recommended - uses proper module entry point): python -m smartem_backend.api_server diff --git a/docs/backend/database.md b/docs/backend/database.md index f5a1600..f098c0d 100644 --- a/docs/backend/database.md +++ b/docs/backend/database.md @@ -35,8 +35,10 @@ This will: 1. Create the Alembic version tracking table 2. Apply migration 001: Create core SmartEM schema baseline (acquisition, grid, gridsquare, micrograph, foilhole, atlas, etc.) 3. Apply migration 002: Add performance indexes for acquisition datetime queries -4. Apply migration 003: Add prediction model tables -5. Apply remaining migrations for quality metrics, training tables, and schema fixes +4. Apply migration 003: Add SciFi robot prediction model tables +5. Apply migration 004: Add quality metric and training tables +6. Apply migration 005: Fix schema drift and sync indexes +7. Apply migration 006: Add suggested acquisition index ### Running Migrations @@ -232,7 +234,7 @@ python -m alembic revision --autogenerate -m "Sync models with database" **Missing baseline schema**: If you encounter errors like "relation 'gridsquare' does not exist" when running migrations: -This indicates that migration 002 is trying to create indexes on tables that don't exist yet. This happens when the baseline schema migration is missing or not properly applied. The fix is to ensure the baseline migration (6e6302f1ccb6) is applied before migration 002. +This indicates that migration 002 is trying to create indexes on tables that don't exist yet. This happens when the baseline schema migration is missing or not properly applied. The fix is to ensure migration 001 (`2025_09_18_1042-001_create_core_smartem_schema_baseline.py`) is applied before migration 002. ```bash # Solution: Run the complete migration sequence from scratch diff --git a/docs/development/generate-docs.md b/docs/development/generate-docs.md index 7bfe97b..ad12325 100644 --- a/docs/development/generate-docs.md +++ b/docs/development/generate-docs.md @@ -1,8 +1,44 @@ # Generate Documentation -## Documentation Generation +## Overview -```sh -tox -e docs -python -m http.server -d build/html +Documentation lives as Markdown files in the `docs/` directory. During build, these are automatically converted to MDX +and copied into the webui for rendering. The webui is deployed to GitHub Pages. + +## How it works + +1. **Source**: Markdown files in `docs/` (this is what you edit) +2. **Sync**: `webui/scripts/generate-mdx-docs.ts` converts `.md` to `.mdx` and copies them into `webui/src/docs/` (gitignored) +3. **Build**: The sync runs automatically as part of `npm run prebuild` and is watched during `npm run dev` +4. **Deploy**: The `deploy-webui.yml` workflow builds and publishes to GitHub Pages on push to `main` when `docs/**` changes + +## Local preview + +```bash +cd webui +npm install +npm run dev ``` + +This starts the Vite dev server with live reload. Changes to `docs/*.md` files are picked up automatically. + +## Building for production + +```bash +cd webui +npm install +npm run build +``` + +The built output goes to `webui/dist/`. + +## Navigation structure + +The `index.md` files in each `docs/` subdirectory contain `toctree` directives that define navigation order and +grouping. The webui reads these during build to generate the sidebar navigation. When adding a new page, add its +filename (without extension) to the relevant `index.md` toctree. + +## Deployment + +Deployment is automatic via the `deploy-webui.yml` GitHub Actions workflow. It triggers on pushes to `main` that +modify files in `webui/`, `core/`, or `docs/`. Manual deployment is also available via `workflow_dispatch`. diff --git a/docs/development/tools.md b/docs/development/tools.md index 6f0939c..5961c0a 100644 --- a/docs/development/tools.md +++ b/docs/development/tools.md @@ -143,6 +143,57 @@ This tool is particularly useful for: Both tools work together to simulate the complete external data flow into the SmartEM system, enabling comprehensive testing without requiring actual microscopy equipment or external processing systems. +## Database Tools + +### Schema Drift Detection + +Detects when SQLModel definitions have changed without corresponding Alembic migrations. Creates a temporary PostgreSQL database, applies existing migrations, then runs autogenerate to check for differences. Also used in CI via the `_schema_drift.yml` workflow. + +```bash +# Requires a running PostgreSQL instance (e.g. from dev-k8s.sh up) +uv run python tools/check_schema_drift.py +``` + +Exit code 0 means the schema is in sync; exit code 1 means drift was detected (or an error occurred). + +### Database Table Row Counts + +Connects to PostgreSQL and outputs row counts for every SmartEM table. Useful for verifying data after E2E tests or checking the state of a development database. + +```bash +uv run python tools/db_table_totals.py +``` + +## API Documentation Generation + +### Generate API Docs + +Generates API documentation from OpenAPI specs into `docs/api/`. Processes two APIs: + +- **Athena API**: copies the original spec from `docs/athena-decision-service-api-spec.json` and adds a local mock server entry +- **SmartEM API**: imports the FastAPI app and extracts the OpenAPI schema at runtime + +```bash +uv run python tools/generate_api_docs.py +``` + +Output is written to `docs/api/athena/swagger.json` and `docs/api/smartem/swagger.json`. + +## Miscellaneous Tools + +### Create ISO Image + +Creates an ISO image from a source directory using `mkisofs` or `genisoimage`. + +```bash +./tools/makeiso.sh [output.iso] + +# Example: create an ISO from a test dataset +./tools/makeiso.sh ./tests/testdata/bi37708-28 dataset.iso +``` + +If the output filename is omitted, it defaults to `.iso`. + ## Additional Development Commands ### Pre-commit Workflow diff --git a/docs/glossary.md b/docs/glossary.md index afe6cea..c930d2b 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -19,9 +19,9 @@ collection strategies. ## ARIA -Automated Real-time Image Analysis system developed by INSTRUCT-ERIC for real-time processing and quality assessment -of cryo-electron microscopy data during acquisition. More information available at: -https://instruct-eric.org/help/about-aria +Central metadata repository for structural biology data from multiple facilities, managed by INSTRUCT-ERIC. ARIA +collects and organises acquisition metadata, sample information, and processing results deposited from facilities via +the FandanGO plugin framework. More information available at: https://instruct-eric.org/help/about-aria ## Atlas diff --git a/docs/operations/index.md b/docs/operations/index.md index d64c374..b9cebdb 100644 --- a/docs/operations/index.md +++ b/docs/operations/index.md @@ -5,6 +5,8 @@ Cross-cutting operational documentation for deploying and running SmartEM in pro ```{toctree} :maxdepth: 1 +releasing +publish-smartem-workspace-to-pypi kubernetes kubernetes-secrets containerization @@ -16,6 +18,11 @@ logging ## Topics +### Releases + +- [Release Procedure](releasing.md) - How to release smartem-decisions, smartem-epuplayer, and smartem-workspace +- [PyPI Token Setup](publish-smartem-workspace-to-pypi.md) - First-time PyPI account and token configuration + ### Kubernetes - [Kubernetes Deployment](kubernetes.md) - Deploying SmartEM to Kubernetes clusters diff --git a/docs/operations/releasing.md b/docs/operations/releasing.md new file mode 100644 index 0000000..839f304 --- /dev/null +++ b/docs/operations/releasing.md @@ -0,0 +1,198 @@ +# Release Procedure + +How to release the three packages in the SmartEM ecosystem. All releases are tag-driven via GitHub Actions. + +## Release types + +| Type | Trigger | Purpose | +|------|---------|---------| +| **RC (Release Candidate)** | Automatic on push to `main` (when path filters match) | Pre-release for testing | +| **Stable** | Push a version tag | Production release | +| **Manual** | `workflow_dispatch` with `rc` or `stable` choice | Emergency or ad-hoc releases | + +## General flow + +1. Develop on a feature branch +2. Merge PR to `main` +3. CI automatically creates an RC release (if path filters match and no stable tag exists for the current version) +4. When ready for stable: push a version tag +5. CI runs tests, builds artifacts, publishes to PyPI/GHCR, and creates a GitHub Release + +## smartem-decisions + +**Repository:** [DiamondLightSource/smartem-decisions](https://github.com/DiamondLightSource/smartem-decisions) +**Workflow:** `release-smartem-decisions.yml` +**Versioning:** `setuptools_scm` (version derived from git tags automatically) +**Tag prefix:** `smartem-decisions-v` + +### Path filters + +The workflow triggers on changes to: +- `src/smartem_agent/**` +- `src/smartem_common/**` +- `src/smartem_backend/api_client.py` + +### Artifacts produced + +- Python wheel (PyPI, stable only) +- Windows executable (`smartem-agent-windows-v{VERSION}.exe`) +- Docker container image (GHCR, stable only) +- GitHub Release with all artifacts + +### Releasing a stable version + +```bash +# Version is determined by setuptools_scm from the tag +git tag smartem-decisions-v1.2.0 +git push origin smartem-decisions-v1.2.0 +``` + +CI will: +1. Run tests on Linux and Windows +2. Lint with ruff and pyright +3. Build Python wheel and Windows exe +4. Smoke test the Windows exe +5. Publish wheel to PyPI (trusted publishing via OIDC) +6. Build and push Docker image to GHCR (`ghcr.io/DiamondLightSource/smartem-decisions:{VERSION}`) +7. Create a GitHub Release with release notes, wheel, and exe attached + +## smartem-epuplayer + +**Repository:** [DiamondLightSource/smartem-devtools](https://github.com/DiamondLightSource/smartem-devtools) +**Workflow:** `release-smartem-epuplayer.yml` +**Versioning:** Manual — set in both `pyproject.toml` and `smartem_epuplayer/__init__.py` (must match) +**Tag prefix:** `epuplayer-v` +**Package directory:** `packages/smartem-epuplayer/` + +### Path filters + +The workflow triggers on changes to: +- `packages/smartem-epuplayer/**` + +### Artifacts produced + +- Python wheel (PyPI, stable only) +- Windows executable (`epuplayer-windows-v{VERSION}.exe`) +- GitHub Release with all artifacts + +### Bumping the version + +Before tagging, update the version in both places: + +```python +# packages/smartem-epuplayer/pyproject.toml +version = "0.3.0" + +# packages/smartem-epuplayer/smartem_epuplayer/__init__.py +__version__ = "0.3.0" +``` + +The CI will fail if these don't match. + +### Releasing a stable version + +```bash +git tag epuplayer-v0.3.0 +git push origin epuplayer-v0.3.0 +``` + +## smartem-workspace + +**Repository:** [DiamondLightSource/smartem-devtools](https://github.com/DiamondLightSource/smartem-devtools) +**Workflow:** `release-smartem-workspace.yml` +**Versioning:** Manual — set in both `pyproject.toml` and `smartem_workspace/__init__.py` (must match) +**Tag prefix:** `smartem-workspace-v` +**Package directory:** `packages/smartem-workspace/` + +### Path filters + +The workflow triggers on changes to: +- `packages/smartem-workspace/**` + +### Artifacts produced + +- Python wheel (PyPI, stable only) +- GitHub Release + +### Bumping the version + +Same pattern as epuplayer — update both files: + +```python +# packages/smartem-workspace/pyproject.toml +version = "0.4.0" + +# packages/smartem-workspace/smartem_workspace/__init__.py +__version__ = "0.4.0" +``` + +### Releasing a stable version + +```bash +git tag smartem-workspace-v0.4.0 +git push origin smartem-workspace-v0.4.0 +``` + +### uvx cache gotcha + +`smartem-workspace` is commonly run via `uvx`. After publishing a new version, users may still get the old version due +to uvx caching. They need to either: + +```bash +# Force refresh the cache +uvx --refresh smartem-workspace --help + +# Or pin to latest explicitly +uvx smartem-workspace@latest --help +``` + +## Manual / emergency releases + +All three workflows support `workflow_dispatch`. Go to the Actions tab of the relevant repository, select the release +workflow, click "Run workflow", and choose `rc` or `stable`. + +This is useful when: +- An RC wasn't triggered automatically (e.g. path filters didn't match) +- You need to re-release after a CI fix +- You want a stable release without pushing a tag first + +## Verifying a release + +### GitHub Releases + +Check the Releases page of the relevant repository. Both RC and stable releases appear there (RCs are marked as +pre-release). + +### PyPI (stable only) + +```bash +# smartem-decisions +pip install smartem-decisions==1.2.0 + +# smartem-epuplayer +pip install smartem-epuplayer==0.3.0 + +# smartem-workspace +pip install smartem-workspace==0.4.0 +``` + +### Docker (smartem-decisions stable only) + +```bash +docker pull ghcr.io/diamondlightsource/smartem-decisions:1.2.0 +``` + +### Windows executable + +Download from the GitHub Release assets and run: + +```bash +smartem-agent-windows-v1.2.0.exe --help +epuplayer-windows-v0.3.0.exe --help +``` + +## First-time PyPI setup + +If you're setting up PyPI publishing for the first time (new package or new repository), see +[PyPI Token Setup](publish-smartem-workspace-to-pypi.md) for account creation, token generation, and GitHub Secrets +configuration.