Add OWN_GIL subinterpreter execution to event loop pool#44
Merged
Conversation
This adds true parallel Python execution with OWN_GIL subinterpreters
while keeping event loop coordination in the main interpreter.
Key features:
- Session management with process affinity (same PID -> same worker)
- Sessions created automatically on first task from a process
- Sessions cleaned up via process monitoring on process exit
- Asyncio event loop per worker for coroutine support
- Tasks run via run_until_complete() in worker's event loop
- Results sent back via enif_send() to calling process
New functions:
- py_nif:owngil_create_session/1 - Create session in worker
- py_nif:owngil_submit_task/7 - Submit async task to worker
- py_nif:owngil_destroy_session/2 - Destroy session
Configuration:
{erlang_python, [{event_loop_pool_owngil, true}]}
When enabled, py_event_loop_pool:create_task/3,4 and spawn_task/3,4
automatically route through OWN_GIL workers instead of the shared
event loop.
Compares regular event loop pool vs OWN_GIL mode: - Sequential calls (single caller) - Concurrent calls (multiple processes) - CPU-bound parallel tasks (time.sleep) Run with: escript examples/bench_owngil_pool.erl
- Use ets:foldl instead of match spec in handle_info DOWN to avoid dialyzer type violations with '_' atoms in record construction - Remove unused destroy_session/2 function - Fix test_async_coroutine_call to use asyncio.sleep instead of custom function (OWN_GIL subinterpreters have separate namespaces) - Fix test_same_process_same_worker to test process affinity without relying on shared namespace state
- Simplify test_async_coroutine_call to use math.sqrt instead of asyncio.sleep (more reliable on CI) - Add OWN_GIL event loop pool documentation to event_loop_architecture.md covering architecture, configuration, usage, and performance
benoitc
added a commit
that referenced
this pull request
Mar 20, 2026
Enable true parallel Python execution with per-worker GILs.
Sessions map PIDs to workers with automatic cleanup on process exit.
Config: {event_loop_pool_owngil, true}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
Erlang:
py_event_loop_pool.erl: Session registry, task routing, process monitoringpy_nif.erl: New NIF exports for session managementC:
py_nif.c: NIFs for create/submit/destroy sessionpy_subinterp_thread.c/h: Asyncio initialization, async task handlingTests:
py_event_loop_pool_owngil_SUITE.erl: 9 tests for OWN_GIL modeConfiguration
{erlang_python, [ {event_loop_pool_size, 4}, {event_loop_pool_owngil, true} ]}When enabled,
py_event_loop_pool:create_task/3,4andspawn_task/3,4automatically route through OWN_GIL workers.