Skip to content

fix: improve handling of file and directory events in FlowsMapper#4031

Merged
reubenmiller merged 3 commits intothin-edge:mainfrom
reubenmiller:fix-flows-inotify-event-sequence
Mar 16, 2026
Merged

fix: improve handling of file and directory events in FlowsMapper#4031
reubenmiller merged 3 commits intothin-edge:mainfrom
reubenmiller:fix-flows-inotify-event-sequence

Conversation

@reubenmiller
Copy link
Copy Markdown
Contributor

@reubenmiller reubenmiller commented Mar 10, 2026

Proposed changes

Fixes flow being permanently lost from memory after a package manager updates it (deletes then recreates the files), even though files are restored on disk (#4023).

Three targeted fixes:

1. remove_script() — unload instead of remove (registry.rs)

When a script file is deleted, flows referencing it are now unloaded (moved to unloaded_flows) instead of being silently ignored with a warning. This means when the script is restored and reload_script() calls drain_unloaded(), the flow is rediscovered and reloaded automatically.

Previously, remove_script() only logged a warning and returned early — the flow stayed in the flows map with a broken script reference but was never re-evaluated when the script came back.

2. add_flow() — no-op on missing file instead of destructive removal (registry.rs)

Changed from if tokio::fs::read_to_string().is_err() → remove_flow() to if !path.is_file() → return.

The old code would destroy a valid in-memory flow on a transient read failure (e.g. file briefly absent during a package update). The new code simply returns — if the file was truly deleted, a FileDeleted event will handle removal; if it's being replaced, a subsequent Modified event will reload it.

3. on_file_removed() — skip stale FileDeleted events (actor.rs)

Added a path.exists() guard at the top of on_file_removed(). During a package update, FileDeleted events can arrive after the file has already been re-created. Without this guard, the stale event would remove the freshly-reloaded flow.

Also added the missing send_updated_subscriptions() + update_flow_status() calls for script removals, which were previously only done for .toml removals.

Types of changes

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
  • Documentation Update (if none of the other choices apply)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

#4023

Checklist

  • I have read the CONTRIBUTING doc
  • I have signed the CLA (in all commits with git commit -s. You can activate automatic signing by running just prepare-dev once)
  • I ran just format as mentioned in CODING_GUIDELINES
  • I used just check as mentioned in CODING_GUIDELINES
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

Further comments

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 10, 2026

Robot Results

✅ Passed ❌ Failed ⏭️ Skipped Total Pass % ⏱️ Duration
858 0 3 858 100 2h41m12.544329s

@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 11, 2026

Codecov Report

❌ Patch coverage is 0% with 16 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
crates/extensions/tedge_flows/src/registry.rs 0.00% 11 Missing ⚠️
crates/extensions/tedge_flows/src/actor.rs 0.00% 5 Missing ⚠️

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread tests/RobotFramework/tests/tedge_flows/flow_lifecycle.robot
Comment thread tests/RobotFramework/tests/tedge_flows/flow_lifecycle.robot
Comment thread crates/extensions/tedge_flows/src/registry.rs
Comment thread crates/extensions/tedge_flows/src/actor.rs
@Bravo555
Copy link
Copy Markdown
Member

Bravo555 commented Mar 11, 2026

FileDeleted(flow.toml) (stale, delayed past the debounce window) → ❌ "Removing flow" — flow removed again from a stale event

I'm not able to observe this when running the Flow is reloaded after package-style replacement test.

I placed the breakpoint at Install Lifecycle Flow main-v1.js line and after running "Debug test" ran the following commands to enable debug log from tedge_flows:

$ systemctl edit tedge-mapper-local

# place these lines in the region indicated by `systemctl edit`
[Service]
Environment=RUST_LOG=debug

$ systemctl restart tedge-mapper-local
$ journalctl -f -u tedge-mapper-local

Then I continued until Simulate package install: step and ran that. These are the messages sent by inotify:

2026-03-11T15:32:21.891735368Z DEBUG Inotify: /workspace/crates/core/tedge_actors/src/message_boxes.rs:212: send Modified("/etc/tedge/mappers/local/flows/lifecycle-test/flow.toml")
2026-03-11T15:32:21.891756037Z DEBUG Inotify: /workspace/crates/core/tedge_actors/src/message_boxes.rs:212: send FileDeleted("/etc/tedge/mappers/local/flows/lifecycle-test/flow.toml")
2026-03-11T15:32:21.891760576Z DEBUG Inotify: /workspace/crates/core/tedge_actors/src/message_boxes.rs:212: send Modified("/etc/tedge/mappers/local/flows/lifecycle-test/main.js")
2026-03-11T15:32:21.891763902Z DEBUG Inotify: /workspace/crates/core/tedge_actors/src/message_boxes.rs:212: send FileDeleted("/etc/tedge/mappers/local/flows/lifecycle-test/main.js")
2026-03-11T15:32:21.891767569Z DEBUG Inotify: /workspace/crates/core/tedge_actors/src/message_boxes.rs:212: send FileCreated("/etc/tedge/mappers/local/flows/lifecycle-test/flow.toml")
2026-03-11T15:32:21.891770926Z DEBUG Inotify: /workspace/crates/core/tedge_actors/src/message_boxes.rs:212: send Modified("/etc/tedge/mappers/local/flows/lifecycle-test/flow.toml")
2026-03-11T15:32:21.891774372Z DEBUG Inotify: /workspace/crates/core/tedge_actors/src/message_boxes.rs:212: send FileCreated("/etc/tedge/mappers/local/flows/lifecycle-test/main.js")
2026-03-11T15:32:21.891777929Z DEBUG Inotify: /workspace/crates/core/tedge_actors/src/message_boxes.rs:212: send Modified("/etc/tedge/mappers/local/flows/lifecycle-test/main.js")

There are FileDeleted events, but there are also FileCreated and Modified events after that, which should cause the flow to be properly reloaded.

Copy link
Copy Markdown
Member

@Bravo555 Bravo555 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixes the problem with reloading scripts, approved

@Bravo555 Bravo555 removed their assignment Mar 16, 2026
@reubenmiller reubenmiller marked this pull request as ready for review March 16, 2026 11:16
Copy link
Copy Markdown
Contributor

@didier-wenzek didier-wenzek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved

Comment on lines 551 to 559
if matches!(path.extension(), Some("js" | "ts" | "mjs")) {
self.processor.remove_script(path).await;
self.send_updated_subscriptions().await?;
self.update_flow_status(path).await?;
} else if path.extension() == Some("toml") {
self.processor.remove_flow(path).await;
self.send_updated_subscriptions().await?;
self.update_flow_status(path).await?;
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if matches!(path.extension(), Some("js" | "ts" | "mjs")) {
    self.processor.remove_script(path).await;
} else if path.extension() == Some("toml") {
    self.processor.remove_flow(path).await;
}
self.send_updated_subscriptions().await?;
self.update_flow_status(path).await?;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess that assumes that we don't get any other notifications which are from .js, .ts, .mjs nor .toml files?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are correct, we have to handle the case of a flows unrelated file.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed the recommended change

reubenmiller and others added 3 commits March 16, 2026 21:01
Signed-off-by: reubenmiller <reuben.d.miller@gmail.com>
…ning

Signed-off-by: reubenmiller <reuben.d.miller@gmail.com>
Signed-off-by: Marcel Guzik <marcel.guzik@cumulocity.com>
@reubenmiller reubenmiller force-pushed the fix-flows-inotify-event-sequence branch from 2c1d75a to a016b7b Compare March 16, 2026 20:02
@reubenmiller reubenmiller temporarily deployed to Test Pull Request March 16, 2026 20:02 — with GitHub Actions Inactive
@reubenmiller reubenmiller added this pull request to the merge queue Mar 16, 2026
Merged via the queue into thin-edge:main with commit 5e73020 Mar 16, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants