[feat] Unify async DKG Publisher setup, management, and runtime fixes#43
[feat] Unify async DKG Publisher setup, management, and runtime fixes#43Jurij89 wants to merge 10 commits intofix/publisher-plugin-error-messagefrom
Conversation
Introduce script:publisher for DKG Publisher plugin config updates, wallet management, and fresh setup reset. Unify setup messaging around async publishing + DKG Publisher plugin and add helper/unit-test coverage for env/config handling.
…publishOptions input Align Publisher vs Essentials tool descriptions to reduce sync/async ambiguity. Expand knowledge-asset-publish controls via publishOptions (privacy/priority/epochs/maxAttempts) without retrieval-tool behavior changes.
Delete packages/plugin-dkg-publisher/setup.js and remove setup entrypoints from plugin package scripts. Keep Agent-owned setup/management as the only supported flow and update legacy bootstrap wording.
Use @bull-board/api/bullMQAdapter.js so async publisher loads under ESM. Align document-to-markdown REST endpoint to /api/document-to-markdown for route consistency.
Add --shims to publisher dev/build scripts so __dirname is defined in dist/*.mjs. Prevents __dirname runtime crashes when async publisher is loaded via ESM imports.
Set runtime default epochs to 12 when publishOptions.epochs is not provided. Align assets table schema default for epochs to 12 for consistency.
Jurij89
changed the title
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
This PR adds substantial async publisher setup and migrates DKG retrieval/query HTTP routes into plugin-dkg-essentials, but it introduces API compatibility blockers. The MCP publish tool contract now drops legacy top-level publish controls, and the HTTP API changes remove/alter previously accepted request shapes without a migration path. Maintainability in touched setup code also worsens due to a new multi-responsibility 500+ line module.
| }) | ||
| .optional(), | ||
| privacy: z.enum(["private", "public"]).optional(), | ||
| publishOptions: z |
There was a problem hiding this comment.
🔴 Bug: Moving publish controls under publishOptions without a backward-compatible alias breaks existing MCP clients that still send top-level privacy/priority/epochs/maxAttempts; those inputs are now ignored and can silently change behavior (e.g., defaulting to private). Accept legacy fields (or fail fast with a clear validation error) before applying defaults.
| body: z.object({ | ||
| query: z.string().min(1, "Query cannot be empty"), | ||
| queryType: z | ||
| .enum(["SELECT", "CONSTRUCT"]) |
There was a problem hiding this comment.
🔴 Bug: Restricting queryType to SELECT | CONSTRUCT drops ASK/DESCRIBE support from the previous /api/dkg/query contract, so existing clients now get 400s. Keep prior query types for compatibility or version this endpoint and document migration.
| ); | ||
|
|
||
| api.get( | ||
| "/api/dkg/get", |
There was a problem hiding this comment.
🔴 Bug: Introducing /api/dkg/get without preserving the previous retrieval route (/api/dkg/assets?ual=...) is a breaking HTTP API change for existing consumers. Add a deprecated compatibility alias returning the same response shape or version the API.
| @@ -0,0 +1,553 @@ | |||
| import path from "path"; | |||
There was a problem hiding this comment.
🟡 Issue: This new file is 500+ lines and mixes env parsing, provisioning, wallet administration, and reset logic, which increases coupling and violates the repo’s DoD guidance for file size/scope. Split it into focused modules (env/config, DB provisioning, wallet operations) and keep this entrypoint orchestration-only.
…te file Expose sync publish over HTTP with the same create logic as dkg-create MCP (including blob id support). Remove accidentally committed apps/agent/original_dkg_node file and add API coverage tests.
There was a problem hiding this comment.
This PR makes substantial setup/runtime improvements for async publishing, but it introduces multiple API-contract regressions that are likely to break existing clients. The largest issues are endpoint path changes without compatibility aliases and MCP/HTTP schema narrowing that removes previously accepted inputs. There is also a status-code correctness issue where user-input errors are returned as 500s. Maintainability in touched areas is mixed: configuration centralization improved, but contract drift across plugins worsened due unversioned interface changes.
| }) | ||
| .optional(), | ||
| privacy: z.enum(["private", "public"]).optional(), | ||
| publishOptions: z |
There was a problem hiding this comment.
🔴 Bug: This changes the knowledge-asset-publish input contract from top-level privacy/priority/epochs/maxAttempts to nested publishOptions, which breaks existing MCP callers. Keep backward compatibility by accepting both shapes (old + new) and mapping old fields into publishOptions with deprecation messaging.
| // REST endpoint for document-to-markdown conversion | ||
| api.post( | ||
| "/document-to-markdown", | ||
| "/api/document-to-markdown", |
There was a problem hiding this comment.
🔴 Bug: Renaming the route to /api/document-to-markdown removes the previous /document-to-markdown path with no compatibility alias, which is an API breaking change for existing clients. Add a backward-compatible alias route (or versioned migration) before removing the old path.
| body: z.object({ | ||
| query: z.string().min(1, "Query cannot be empty"), | ||
| queryType: z | ||
| .enum(["SELECT", "CONSTRUCT"]) |
There was a problem hiding this comment.
🔴 Bug: Restricting queryType to SELECT | CONSTRUCT is a breaking change for callers that previously used ASK/DESCRIBE on /api/dkg/query. Preserve prior accepted values (or introduce a versioned endpoint) to avoid contract breakage.
| ); | ||
|
|
||
| api.get( | ||
| "/api/dkg/get", |
There was a problem hiding this comment.
🔴 Bug: Introducing GET /api/dkg/get while removing the prior GET /api/dkg/assets endpoint breaks existing HTTP consumers that query assets by UAL. Add a compatibility handler for the old path (or keep both paths during migration).
| } catch (error) { | ||
| const errorMessage = | ||
| error instanceof Error ? error.message : String(error); | ||
| return res.status(500).json({ |
There was a problem hiding this comment.
🔴 Bug: This catch-all returns 500 for input-originated errors (e.g., invalid/missing blob id from resolveJsonLdInput), which misclassifies client errors as server failures. Map validation/input failures to 4xx (400/404) and reserve 500 for unexpected internal errors.
1 participant
Summary
Unified async publishing setup in Agent
The main Agent setup flow now optionally enables the DKG Publisher plugin (recommended path), reuses Engine-provided MySQL credentials, applies sane MySQL/Redis defaults, and avoids duplicate Publisher setup questions.
Publisher runtime gating + single env source
Publisher is loaded conditionally via
ASYNC_PUBLISHING_ENABLED=true, with consolidated config sourced fromapps/agent/.env(legacy.env.publisherfallback remains for compatibility where relevant).Publisher management command (post-setup)
Added a dedicated
script:publishermanagement flow in Agent for updating async publishing config, managing wallets, and running fresh setup/reset behavior.MCP tool clarity + publish input improvements
Publisher MCP descriptions were clarified to avoid sync/async ambiguity with Essentials.
knowledge-asset-publishnow uses canonicalpublishOptionsinput (privacy,priority,epochs,maxAttempts) and defaults privacy to private if omitted.Publisher runtime stability fixes
Fixed ESM import/runtime issues by switching Bull Board adapter import to
@bull-board/api/bullMQAdapter.jsand enabling tsup--shimsfor Publisher build/dev scripts.Legacy setup cleanup
Removed the old
packages/plugin-dkg-publisher/setup.jspath and related script references so Agent setup/management is the single supported setup surface.Additional consistency updates
Updated document-to-markdown endpoint to
/api/document-to-markdownand set Publisher default epochs to 12 when not explicitly provided.Files changed
apps/agent/src/server/scripts/setup.tsapps/agent/src/server/scripts/publisher.tsapps/agent/src/server/setupPublisher.tsapps/agent/src/server/index.tsASYNC_PUBLISHING_ENABLEDpackages/plugin-dkg-publisher/src/config.tspackages/plugin-dkg-publisher/src/index.tspackages/plugin-dkg-publisher/src/mcp/tools.tspublishOptionssupportpackages/plugin-dkg-publisher/src/services/QueueService.tsbullMQAdapter.js)packages/plugin-dkg-publisher/src/services/DkgService.tspackages/plugin-dkg-publisher/src/services/AssetService.tspackages/plugin-dkg-publisher/src/database/schema.tspackages/plugin-dkg-publisher/package.json--shimsfor ESM/CJS runtime compatibilitypackages/plugin-dkg-publisher/setup.jspackages/plugin-dkg-essentials/src/plugins/dkg-tools.tspackages/plugin-dkg-essentials/src/plugins/document-to-markdown/index.ts/api/document-to-markdownTest plan
cd apps/agent && npm run script:setupand verify async publishing opt-in prompt and consolidated flowASYNC_PUBLISHING_ENABLED=truecd apps/agent && npm run script:publisherand validate:knowledge-asset-publishwithpublishOptionsand verify queuing succeedsknowledge-asset-publishdefaults toprivacy=privatewhen omittedpublishOptions.epochsis not providedbullMQAdapterESM import suffix__dirnamein ESM runtime/api/document-to-markdownworks end-to-end and old non-/apipath is not used in clients/tests