Fix segmentation fault in libev prepare_callback during shutdown#708
Open
dkropachev wants to merge 2 commits intomasterfrom
Open
Fix segmentation fault in libev prepare_callback during shutdown#708dkropachev wants to merge 2 commits intomasterfrom
dkropachev wants to merge 2 commits intomasterfrom
Conversation
This commit fixes a race condition that causes segmentation faults when the Python driver shuts down while libev callbacks are still executing. Changes: 1. Add null checks in prepare_callback() to handle destroyed objects 2. Stop prepare watcher before cleanup to prevent race conditions 3. Increase thread join timeout to allow proper shutdown synchronization The issue occurred when metrics cleanup during shutdown raced with libev callbacks executing in background threads, causing access to freed Python objects. Fixes crash at address 0x2f998 in prepare_callback accessing destroyed libevwrapper_Prepare objects.
This extends the segmentation fault fix to cover all libev callbacks that access Python objects during shutdown. Changes: 1. Add null checks in io_callback() before accessing self->callback 2. Add null checks in timer_callback() before accessing self->callback These prevent race conditions similar to the prepare_callback issue where libev callbacks could execute after Python objects are destroyed during driver shutdown.
|
@dkropachev that replace the fix copilot was trying todo in #680 ? on c side of things ? that was reverted ? |
Collaborator
Author
No, it addresses same problem from other side. It compliments it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a race condition that causes segmentation faults when the Python driver shuts down while libev callbacks are still executing.
Addresses: scylladb/scylla-cluster-tests#11713 and #524
Root Cause
Race condition during shutdown where metrics cleanup in main thread races with libev callbacks executing in background thread, leading to access of freed Python objects at address 0x2f998.
Changes
prepare_callback()to handle destroyed objects safelyTest plan