feat: custom cloud mappers by jarhodes314 · Pull Request #4009 · thin-edge/thin-edge.io

jarhodes314 · 2026-02-24T18:23:04Z

Proposed changes

In addition to the openspec proposal, this PR adds a custom cloud mapper that can be run using tedge-mapper custom (the idea being that the user names their custom cloud implementation and then runs this like any other mapper).

The mapper reads its config from a mapper config directory (e.g. /etc/tedge/mappers/thingsboard for tedge-mapper thingsboard). You can (optionally) connect a built-in bridge by adding a mapper.toml file:

url = "127.0.0.1:8883"

You can then add bridge config templates to the bridge/ subdirectory and flows to the flows/ subdirectory.

Still TODO

Ability to disable TLS for the bridge connection (i.e. connect to insecure cloud brokers)
Handle profiles somewhat gracefully in tedge mapper/tedge bridge subcommands (currently they are totally unsupported)
Basic tedge connect --test support
Configurable bridge rules for aws/az mappers
Improve relative path support - tedge mapper config get will currently display the naively generated path including any ..
Consider renaming ${config.} to ${tedge.} in bridge templates
Disable mapper.toml without bridge directory warning for built-in mappers

Proposal

A proposal for how to implement a generic mapper using Openspec. In short, openspec is an IMO very useful tool for making design decisions with AI agents that can later serve as a source of truth.

The first commit is a bunch of files generated by openspec to make it actually work. It's probably worth some discussion about whether we think it's a useful tool.

The (at the time of writing, only) other commit is the actual proposal and design docs for implementing a generic mapper that can tie together the flows and configurable bridge. I would suggest reading it in the order proposal.md, design.md then the two spec.mds.

Types of changes

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
Documentation Update (if none of the other choices apply)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

Checklist

I have read the CONTRIBUTING doc
I have signed the CLA (in all commits with git commit -s. You can activate automatic signing by running just prepare-dev once)
I ran just format as mentioned in CODING_GUIDELINES
I used just check as mentioned in CODING_GUIDELINES
I have added tests that prove my fix is effective or that my feature works
I have added necessary documentation (if appropriate)

Further comments

codecov · 2026-02-24T21:09:00Z

Codecov Report

❌ Patch coverage is 92.72611% with 341 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
crates/core/tedge_mapper/src/custom/resolve.rs	93.41%	44 Missing and 32 partials ⚠️
crates/core/tedge_mapper/src/custom/mapper.rs	88.14%	53 Missing and 20 partials ⚠️
crates/core/tedge/src/cli/mapper/cli.rs	91.64%	52 Missing and 5 partials ⚠️
...ge_mqtt_bridge/src/config_toml/parsing/template.rs	91.75%	20 Missing and 3 partials ⚠️
crates/core/tedge/src/cli/bridge/common.rs	77.65%	13 Missing and 8 partials ⚠️
...xtensions/tedge_mqtt_bridge/src/config_toml/mod.rs	96.48%	15 Missing and 1 partial ⚠️
...tedge_mqtt_bridge/src/config_toml/mapper_config.rs	77.77%	12 Missing ⚠️
crates/core/tedge_mapper/src/lib.rs	79.62%	11 Missing ⚠️
...dge_config/src/tedge_toml/tedge_config_location.rs	92.68%	4 Missing and 5 partials ⚠️
crates/core/tedge/src/cli/bridge/test_command.rs	96.35%	5 Missing and 2 partials ⚠️
... and 10 more

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

albinsuresh · 2026-02-25T04:28:54Z

+
+Custom mapper config is isolated — only the custom mapper itself uses it, unlike the global `tedge_config` schema which is referenced throughout thin-edge. This suggests custom mapper settings should live in the mapper's own `tedge.toml` (e.g. `/etc/tedge/mappers/_thingsboard/tedge.toml`), separate from the `define_tedge_config!` macro. Users edit this file directly rather than using `tedge config set/get`.
+
+This avoids the complexity of fitting dynamic/unknown mapper names into the compile-time config macro, and is acceptable because the users writing custom mappers are technical enough to edit TOML files. The bridge template system would need a new `${mapper.*}` variable namespace to let bridge rule templates reference the mapper's own config (analogous to how `${config.c8y.url}` references global config today). `tedge config` support could potentially be added later if demand warrants it (though this might prove complex to implement in practice).


IIRC, the primary use-case for tedge config over editing these toml files by hand was to enable easier iterative updates to the config from automation scripts. So, I wouldn't rule out its utility.

github-actions · 2026-02-26T17:40:48Z

Robot Results

✅ Passed	❌ Failed	⏭️ Skipped	Total	Pass %	⏱️ Duration
894	0	3	894	100	2h14m7.402502999s

albinsuresh · 2026-03-05T16:37:59Z

+use tedge_config::models::HostPort;
+use tedge_config::models::MQTT_TLS_PORT;
+
+/// Parsed custom mapper `tedge.toml` configuration.


Since the contents of this file are read from the mapper.* namespace, wondering if naming the config file mapper.toml instead of tedge.toml would make it easier for the users even us to easily do that association in our heads. I might even go one step further and change the config.* namespace into `tedge.*.

I have now renamed tedge.toml to mapper.toml following discussion with @reubenmiller on Friday

I might even go one step further and change the config.* namespace into `tedge.*.

That's an interesting thought, I'll give it some consideration

Signed-off-by: James Rhodes <jarhodes314@gmail.com>

…en deciding if mapper.toml is empty Signed-off-by: James Rhodes <jarhodes314@gmail.com>

…t flows-only mappers

Signed-off-by: James Rhodes <jarhodes314@gmail.com>

- Make cert CN take precedence over manually configured device.id, like we do for other clouds - Make device.id required for a custom mapper bridge connection Signed-off-by: James Rhodes <jarhodes314@gmail.com>

Signed-off-by: James Rhodes <jarhodes314@gmail.com>

${mapper.*} variables in bridge rules were expanded against an empty table, causing them to always produce empty strings. The fix threads the fully resolved mapper configuration through to the rule expansion step for both built-in (c8y) and custom mappers. To support this, EffectiveMapperConfig gains two new methods: - `to_template_table()`: builds the TOML table used for ${mapper.*} expansion, overlaying schema-resolved values (cert CN, tedge.toml fallbacks) onto the raw mapper.toml table so non-schema keys like bridge.topic_prefix are also reachable - `get(key) -> ConfigGetResult`: source-aware key lookup distinguishing Value / NotSet / UnknownKey, used by `tedge mapper config get` `resolve_effective_config` gains two new parameters: an optional overlay table (built-in mappers pass their full serialised config so CLI commands reflect real runtime values) and an optional JSON schema override (to distinguish NotSet from UnknownKey for keys absent from the TOML). A new `resolve_effective_mapper_config` function on the c8y mapper serialises the full C8y reader as a base overlay and JSON schema, shared between bridge rule loading and the CLI. `tedge mapper config get` is rewritten to use the new API: it handles built-in and custom mappers uniformly, produces precise error messages ("Key not set" vs "Unknown key", with a cert-specific hint for device.id), and prints a `tedge config set` suggestion for built-in mapper keys. The execute body is extracted into `run(out, err)` so tests can assert on the actual printed output and stderr annotations. `cloud_type` in mapper.toml is now parsed as `tedge_config::models::CloudType` rather than a free-form string, so unknown values are a hard error at parse time. A new startup warning is emitted for mapper directories that contain a `bridge/` subdirectory but no `mapper.toml` — these mappers will always fail to start and the warning points the user at the fix. `resolve_cloud` is deduplicated into cli::common, shared by bridge and mapper CLI code. Signed-off-by: James Rhodes <jarhodes314@gmail.com>

Signed-off-by: James Rhodes <jarhodes314@gmail.com>

reubenmiller

Really nice PR and brings some really exciting features

jarhodes314 requested review from albinsuresh, didier-wenzek and reubenmiller as code owners February 24, 2026 18:23

jarhodes314 temporarily deployed to Test Pull Request February 24, 2026 18:23 — with GitHub Actions Inactive

albinsuresh reviewed Feb 25, 2026

View reviewed changes

didier-wenzek reviewed Feb 25, 2026

View reviewed changes

jarhodes314 force-pushed the proposal/generic-mapper branch from 96b1101 to 80e8738 Compare February 26, 2026 15:34

jarhodes314 requested a review from rina23q as a code owner February 26, 2026 15:34

jarhodes314 had a problem deploying to Test Pull Request February 26, 2026 15:34 — with GitHub Actions Failure

jarhodes314 had a problem deploying to Test Pull Request February 26, 2026 16:37 — with GitHub Actions Failure

jarhodes314 force-pushed the proposal/generic-mapper branch from f04ef32 to 90ce6c9 Compare February 26, 2026 16:37

jarhodes314 temporarily deployed to Test Pull Request February 26, 2026 16:38 — with GitHub Actions Inactive

jarhodes314 had a problem deploying to Test Auto February 26, 2026 17:16 — with GitHub Actions Failure

jarhodes314 changed the title ~~Proposal for generic/custom cloud mapper~~ Initial attempt at a generic/custom cloud mapper Feb 26, 2026

albinsuresh reviewed Mar 5, 2026

View reviewed changes

jarhodes314 had a problem deploying to Test Pull Request March 6, 2026 12:27 — with GitHub Actions Failure

jarhodes314 had a problem deploying to Test Pull Request March 6, 2026 13:57 — with GitHub Actions Failure

jarhodes314 requested a review from a team as a code owner March 6, 2026 16:18

jarhodes314 temporarily deployed to Test Pull Request March 6, 2026 16:18 — with GitHub Actions Inactive

jarhodes314 had a problem deploying to Test Auto March 6, 2026 16:27 — with GitHub Actions Failure

jarhodes314 had a problem deploying to Test Pull Request March 6, 2026 18:26 — with GitHub Actions Failure

jarhodes314 had a problem deploying to Test Pull Request March 9, 2026 14:41 — with GitHub Actions Failure

jarhodes314 had a problem deploying to Test Pull Request March 12, 2026 12:03 — with GitHub Actions Failure

jarhodes314 had a problem deploying to Test Pull Request March 18, 2026 12:00 — with GitHub Actions Failure

jarhodes314 force-pushed the proposal/generic-mapper branch from 4b2bff6 to 0e2746a Compare March 18, 2026 12:17

jarhodes314 temporarily deployed to Test Pull Request March 18, 2026 12:17 — with GitHub Actions Inactive

jarhodes314 had a problem deploying to Test Auto March 18, 2026 12:27 — with GitHub Actions Failure

jarhodes314 had a problem deploying to Test Pull Request March 18, 2026 15:25 — with GitHub Actions Failure

jarhodes314 added 22 commits March 26, 2026 16:41

Document and system test custom mapper

8961064

Signed-off-by: James Rhodes <jarhodes314@gmail.com>

Rename service from tedge-mapper@<name> to tedge-mapper-<name>

b738046

Signed-off-by: James Rhodes <jarhodes314@gmail.com>

Fix lint errors

8190ac6

Signed-off-by: James Rhodes <jarhodes314@gmail.com>

Archive completed specs

1f4017d

Signed-off-by: James Rhodes <jarhodes314@gmail.com>

Don't warn about flows-only mappers

08a8479

Signed-off-by: James Rhodes <jarhodes314@gmail.com>

Ensure cloud_type is set for migrated profiled configs and ignored wh…

6fa43a7

…en deciding if mapper.toml is empty Signed-off-by: James Rhodes <jarhodes314@gmail.com>

Correct documentation line break and allow tedge mapper list to spo…

9fd465f

…t flows-only mappers

Allow ${mapper. syntax in all bridges, not just custom mappers

ff6b1fd

Signed-off-by: James Rhodes <jarhodes314@gmail.com>

Create flows dir on custom mapper startup

cabb314

Signed-off-by: James Rhodes <jarhodes314@gmail.com>

Format tedge mapper list output to make it easier to parse

6593561

Signed-off-by: James Rhodes <jarhodes314@gmail.com>

Adjust device id requirements for custom mapper bridge

ca7a2b9

- Make cert CN take precedence over manually configured device.id, like we do for other clouds - Make device.id required for a custom mapper bridge connection Signed-off-by: James Rhodes <jarhodes314@gmail.com>

Simplify design/proposal/tasks into single doc

a956cfc

Signed-off-by: James Rhodes <jarhodes314@gmail.com>

Add tedge bridge cli support for the custom mappers

b26a448

Signed-off-by: James Rhodes <jarhodes314@gmail.com>

Improve tedge mapper subcommand with single source of truth config

2152151

Signed-off-by: James Rhodes <jarhodes314@gmail.com>

Improve docs

78995b3

Signed-off-by: James Rhodes <jarhodes314@gmail.com>

Run formatter and fix clippy issues

d0db7bf

Signed-off-by: James Rhodes <jarhodes314@gmail.com>

Simplify tedge mapper list output for easier post-processing

34b7f95

Signed-off-by: James Rhodes <jarhodes314@gmail.com>

Improve tedge_mqtt_bridge error messages

08a74df

Signed-off-by: James Rhodes <jarhodes314@gmail.com>

Refactor tedge config set help text to shared location

47ecf2d

Signed-off-by: James Rhodes <jarhodes314@gmail.com>

Fix permissions of mapper directory in custom-mapper system tests

f8211fa

Signed-off-by: James Rhodes <jarhodes314@gmail.com>

Fix mapper warnings and ensure mapper config get url works

6c75d45

Signed-off-by: James Rhodes <jarhodes314@gmail.com>

jarhodes314 force-pushed the proposal/generic-mapper branch from f4ee313 to 6c75d45 Compare March 26, 2026 16:41

jarhodes314 temporarily deployed to Test Pull Request March 26, 2026 16:41 — with GitHub Actions Inactive

jarhodes314 enabled auto-merge March 26, 2026 16:41

jarhodes314 temporarily deployed to Test Auto March 26, 2026 16:50 — with GitHub Actions Inactive

reubenmiller approved these changes Mar 26, 2026

View reviewed changes

jarhodes314 added this pull request to the merge queue Mar 26, 2026

Merged via the queue into thin-edge:main with commit d8dd6a9 Mar 26, 2026
34 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: custom cloud mappers#4009

feat: custom cloud mappers#4009
jarhodes314 merged 35 commits intothin-edge:mainfrom
jarhodes314:proposal/generic-mapper

jarhodes314 commented Feb 24, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Feb 24, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

albinsuresh Feb 25, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Feb 26, 2026 •

edited

Loading

Uh oh!

albinsuresh Mar 5, 2026

Uh oh!

jarhodes314 Mar 9, 2026

Uh oh!

reubenmiller left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants


		Custom mapper config is isolated — only the custom mapper itself uses it, unlike the global `tedge_config` schema which is referenced throughout thin-edge. This suggests custom mapper settings should live in the mapper's own `tedge.toml` (e.g. `/etc/tedge/mappers/_thingsboard/tedge.toml`), separate from the `define_tedge_config!` macro. Users edit this file directly rather than using `tedge config set/get`.

		This avoids the complexity of fitting dynamic/unknown mapper names into the compile-time config macro, and is acceptable because the users writing custom mappers are technical enough to edit TOML files. The bridge template system would need a new `${mapper.*}` variable namespace to let bridge rule templates reference the mapper's own config (analogous to how `${config.c8y.url}` references global config today). `tedge config` support could potentially be added later if demand warrants it (though this might prove complex to implement in practice).

Conversation

jarhodes314 commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Still TODO

Proposal

Types of changes

Paste Link to the issue

Checklist

Further comments

Uh oh!

codecov Bot commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

albinsuresh Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Robot Results

Uh oh!

albinsuresh Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

jarhodes314 Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

reubenmiller left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jarhodes314 commented Feb 24, 2026 •

edited

Loading

codecov Bot commented Feb 24, 2026 •

edited

Loading

github-actions Bot commented Feb 26, 2026 •

edited

Loading