feat: add plugins REST API, automation planner, and adversary schema enhancements#3337
Draft
deacon-mp wants to merge 30 commits into
Draft
feat: add plugins REST API, automation planner, and adversary schema enhancements#3337deacon-mp wants to merge 30 commits into
deacon-mp wants to merge 30 commits into
Conversation
* updating submodule * updating abilities
Contributor
There was a problem hiding this comment.
Pull request overview
This draft PR introduces a new lazy plugin management flow (with a v2 plugins API and a PluginManager), adds an automation import service and planner, and expands adversary atomic ordering to support per-step metadata (including fact injection), alongside various config/dependency and frontend bundle updates.
Changes:
- Added
PluginManager+ new v2 plugin endpoints for enabling/disabling plugins and rebuilding the GUI with restart signaling. - Extended adversary
atomic_orderingto accept structured steps with metadata and updated link generation / atomic planner selection accordingly. - Added automation YAML import service and an automation planner; updated deps/configs and refreshed minified static assets.
Reviewed changes
Copilot reviewed 53 out of 61 changed files in this pull request and generated 15 comments.
Show a summary per file
| File | Description |
|---|---|
| static/js/shared.min.js | Updated shared frontend bundle (REST helper + utilities). |
| static/js/core.min.js | Updated core frontend bundle (health check/version fetch). |
| static/js/ability.min.js | Updated ability UI bundle (filters/search). |
| static/css/timeline.min.css | Updated bundled timeline styles. |
| static/css/shared.min.css | Updated bundled shared styles. |
| static/css/multi-select.min.css | Updated bundled multi-select styles. |
| static/css/modal.min.css | Updated bundled modal styles. |
| static/css/file-explorer.min.css | Updated bundled file-explorer styles. |
| static/css/core.min.css | Updated bundled core layout/navigation styles. |
| static/css/basic.min.css | Updated bundled basic UI styles. |
| server.py | Wires in PluginManager at startup and adjusts plugin init flow. |
| requirements.txt.backup | Adds a backup snapshot of prior Python deps. |
| requirements.txt | Updates/expands Python dependency pins (incl. PyYAML bump). |
| plugins/stockpile | Removes git submodule pointer entry. |
| plugins/ssl | Removes git submodule pointer entry. |
| plugins/sandcat | Removes git submodule pointer entry. |
| plugins/response | Removes git submodule pointer entry. |
| plugins/manx | Removes git submodule pointer entry. |
| plugins/magma | Updates magma submodule pointer entry. |
| plugins/human | Removes git submodule pointer entry. |
| plugins/gameboard | Removes git submodule pointer entry. |
| plugins/fieldmanual | Removes git submodule pointer entry. |
| plugins/emu | Removes git submodule pointer entry. |
| plugins/debrief | Removes git submodule pointer entry. |
| plugins/compass | Removes git submodule pointer entry. |
| plugins/builder | Removes git submodule pointer entry. |
| plugins/atomic | Removes git submodule pointer entry. |
| plugins/access | Removes git submodule pointer entry. |
| package.json | Adds uuid dependency. |
| conf/payloads.yml | Populates payload registry and extension handler mapping. |
| conf/default.yml | Changes default plugin enablement + adds restart flag. |
| conf/agents.yml | Updates agent defaults (deployments + sleep ranges). |
| app/utility/plugin_manager.py | New lazy plugin loading/enabling + GUI build/install helpers. |
| app/service/planning_svc.py | Supports structured atomic steps + metadata-based fact injection into commands. |
| app/service/file_svc.py | Ensures directories exist before saving files. |
| app/service/data_svc.py | Adds safer restore behavior when plugins/modules are missing. |
| app/service/automation_svc.py | New YAML import/validation service for automated operations. |
| app/service/app_svc.py | Adds template setup + discovered plugin registration. |
| app/planners/automation_planner.py | New planner to replay action sequences from facts/relationships. |
| app/planners/atomic.py | Updates atomic selection logic to use step_idx when present. |
| app/objects/secondclass/c_link.py | Adds metadata field support to Link schema/object. |
| app/objects/c_plugin.py | Adds required plugin list + config-driven enable logic. |
| app/objects/c_adversary.py | Allows structured atomic_ordering steps + adds debug logging. |
| app/contacts/contact_tcp.py | Makes manx Session import optional. |
| app/contacts/contact_http.py | Adds exception logging for malformed beacons. |
| app/api/v2/managers/operation_api_manager.py | Minor change during operation setup (stores adversary in a local var). |
| app/api/v2/managers/config_api_manager.py | Fixes plugin config update for plugin vs plugins. |
| app/api/v2/managers/base_api_manager.py | Reworks on-disk update/replace flow; adds extensive debug logging. |
| app/api/v2/managers/adversary_api_manager.py | Adds logging/traceback details around adversary verification. |
| app/api/v2/handlers/plugins_api.py | Adds plugin enable/disable/build-status endpoints + restart behavior. |
| app/api/v2/handlers/health_api.py | Returns 503 “restarting” when restart flag set. |
| app/api/v2/handlers/base_object_api.py | Tightens duplicate-ID check; changes update-on-disk error handling/logging. |
| app/api/v2/handlers/adversary_api.py | Refactors handler style + custom update flow using raw payload. |
| app/api/rest_api.py | Adds a REST enable-plugin handler (route wiring not shown). |
| AdversarySCHEMA.yml | Adds an example adversary schema showing per-step metadata. |
| .gitmodules | Removes plugin submodule definitions. |
| .gitignore | Expands ignores (data/adversaries, mlruns, plugin dirs, coverage). |
| .coveragerc | Removes coverage configuration file. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| module: sys | ||
| type: python_module | ||
| version: 3.9.0 | ||
| restarting: true |
Comment on lines
+80
to
+98
| # ✅ persist enabled plugin in runtime config | ||
| BaseWorld.set_config( | ||
| name="main", | ||
| prop="plugins", | ||
| value=enabled_plugins | ||
| ) | ||
| if build_gui: | ||
| # ✅ mark runtime state (NOT persisted) | ||
| BaseWorld.set_config( | ||
| name="main", | ||
| prop="restarting", | ||
| value=True | ||
| ) | ||
|
|
||
| try: | ||
| await app_svc._save_configurations() | ||
|
|
||
| except Exception as e: | ||
| print(f"Error saving configurations: {e}") |
Comment on lines
+177
to
+188
| async def _disable_build_restart(self, plugin_manager, remaining_plugins): | ||
| await asyncio.sleep(0.5) | ||
|
|
||
| print("[plugin_manager] rebuilding GUI after plugin disable") | ||
|
|
||
| try: | ||
| # reuse existing build pipeline | ||
| await plugin_manager._build_plugin_gui_if_needed("magma") | ||
| except Exception: | ||
| import traceback | ||
| traceback.print_exc() | ||
| return |
Comment on lines
19
to
+25
| def add_routes(self, app: web.Application): | ||
| router = app.router | ||
| router.add_get('/plugins', self.get_plugins) | ||
| router.add_get('/plugins/{name}', self.get_plugin_by_name) | ||
| router.add_post('/plugins/{name}/enable', self.enable_plugin) | ||
| router.add_post('/plugins/disable', self.disable_plugins) | ||
| router.add_get('/plugins/build-status', self.build_status) |
Comment on lines
+25
to
+29
| if BaseWorld.get_config(prop="restarting"): | ||
| return web.json_response( | ||
| {"status": "restarting"}, | ||
| status=503 | ||
| ) |
Comment on lines
+23
to
+25
| atomic_ordering = ma.fields.List( | ||
| ma.fields.Raw(), # Accepts either str or dict — we'll validate in post_load | ||
| ) |
| if os.path.exists(file_path): | ||
| raise JsonHttpBadRequest(f'{self.description.capitalize()} with given id already exists: {obj_id}') | ||
| else: | ||
| self.log.warning('[BaseObjectApi] Adversary found in memory but missing on disk: %s', file_path) |
|
|
||
| except Exception as e: | ||
| self.log.exception('[update_on_disk_object] Exception occurred: %s', str(e)) | ||
| raise web.HTTPInternalServerError(reason='Internal error during adversary update') |
Comment on lines
45
to
+49
| self.app_svc.application.router.add_route('GET', '/file/download_exfil', self.download_exfil_file) | ||
| self.app_svc.application.router.add_route('GET', '/{tail:(?!plugin/|api/v2/).*}', self.handle_catch) | ||
|
|
||
| @check_authorization | ||
| async def enable_plugin(self, request): |
|
|
||
| if links_to_use: | ||
| # Each agent will run the next available step. | ||
| print(f'[Atomic] Running {len(links_to_use)} links with links_to_use: {links_to_use}') |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Work-in-progress feature branch adding:
GET/POST/DELETE /api/v2/pluginsREST API endpoints for plugin managementPluginmodel with enabled/disabled toggle via APIautomation_planner.py- new planner for automated operation executionStatus
Draft PR - needs review and test coverage before merge.