Experiment: fork parallel workers via pcntl_fork()#5663
Conversation
|
/cc @staabm @xificurk Look at this, we might not need the optimizations anymore (unless running on Windows or with OPCache/JIT). It feels like we even need the opposite than before. @staabm Feel free to try out with the process forking performs with and without your LazySourceLocator. I think that the forking might perform better without the introduced laziness (because the memory gets populated in the main process and pcntl_fork just copies (on-write so no memory wasted) the result into the child processes. |
I tried with and without LazySourceLocator but it seem not to make a difference.
in my benchmarking running PHPStan on phpstan-src with forking is 0,5-1 (3-5%) seconds faster I did pcntl_fork experiements with PHPUnit in the past, which also showed creating new processes with forking is faster than bootstrapping new ones from scratch |
| ], | ||
| ]); | ||
|
|
||
| $resultCache = $resultCacheManager->restore($inceptionFiles, false, false, $projectConfigArray, $errorOutput); |
There was a problem hiding this comment.
why do we need the restore after the fork?
can't we restore the cache before forking in the main process?
There was a problem hiding this comment.
This is a rewrite which goal is to make it work 1:1 as before. This code is related to PHPStan Pro. This process can run the whole day. Which means the child always has to get the fresh result cache.
Splits the parallel worker into a process-creation-agnostic half (ProcessBase: the TCP/NDJSON connection and timeout timer) and the process-creation half. The Process interface is what ParallelAnalyser and ProcessPool talk to; the existing react/child-process implementation is renamed to SpawnedProcess. Sole implementation for now — a pcntl_fork() based one follows. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Moves everything WorkerCommand does after CommandHelper::begin() — the TCP connect, NDJSON handshake and per-file analysis loop — into a reusable WorkerRunner service. WorkerCommand now only boots and delegates. This makes the post-boot worker logic callable without a re-boot, which the pcntl_fork() path will rely on. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When PHPSTAN_PARALLEL_FORK=1 is set and OPcache + JIT are off, ParallelAnalyser forks workers from the already-booted process (ForkedProcess) instead of spawning fresh ones — the fork inherits the DI container, skipping the per-worker re-boot. Forked workers still speak the same TCP + NDJSON protocol. ForkParallelChecker gates the path; the full analysed-files list is now threaded through ParallelAnalyser::analyse() so the forked child can set it on NodeScopeResolver — its two callers (AnalyserRunner, FixerWorkerCommand) are updated accordingly. ext-pcntl/ext-posix added to composer.json suggest. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ProcessPromise becomes an interface describing the fixer:worker process as seen by FixerApplication; the existing react/child-process implementation is renamed to SpawnedProcessPromise. FixerApplication creates it through a factory method. Sole implementation for now — a pcntl_fork() based one follows. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Moves everything FixerWorkerCommand does after CommandHelper::begin() — connecting back to FixerApplication, restoring the result cache, running the analyser and streaming results — into a reusable FixerWorkerRunner service. FixerWorkerCommand now only boots and delegates, making the post-boot logic callable without a re-boot. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When PHPSTAN_PARALLEL_FORK=1 is set and OPcache + JIT are off, FixerApplication forks the fixer:worker from the already-booted process (ForkedProcessPromise) instead of spawning a fresh one — the fork inherits the DI container, skipping the re-boot. The forked child still talks to FixerApplication over the same TCP + NDJSON protocol. FixerApplication::run() now receives the InceptionResult so the forked child has the analysed files, project config and error output without re-deriving them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ForkParallelChecker now implements DiagnoseExtension (like Scheduler): under -vvv it prints whether parallel workers are spawned or forked, and — if PHPSTAN_PARALLEL_FORK=1 is set but the fork path still wasn't taken — it explains which precondition (pcntl/posix availability or OPcache/ JIT being off) is missing. The ad-hoc "Note: using pcntl_fork()…" lines that ParallelAnalyser and FixerApplication used to write to stderr in verbose mode are removed in favour of this. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ondrejmirtes
force-pushed
the
pcntl-fork-experiment
branch
from
d876c0e to
9e6a5cd
Compare
|
Should we have a CI job with the env var for test coverage? |
|
Just learning this doesn't work at all with PHARs, because the PHAR is a shared resource and reading source code from it breaks. Needs a different approach. |
2 participants
What
Experimental alternative to the parallel-analysis worker model. Today each
parallel worker is a freshly spawned PHP process (
react/child-process) thatre-boots the whole application — rebuilds the DI container, re-runs bootstrap
files — via
CommandHelper::begin(). That boot is pure overhead, repeated perworker.
This adds a second path: when opted in, workers are
pcntl_fork()-ed from thealready-booted process, inheriting the DI container for free and skipping the
re-boot. Forked workers still talk to the main process over the exact same
TCP + NDJSON protocol — only the process-creation mechanism changes.
Applied to both worker flows:
ParallelAnalyser→worker(the analysis workers)FixerApplication→fixer:worker(the PHPStan Pro worker)Commits
Refactorings first (interfaces with a single implementation), forking
implementations after, ParallelAnalyser and FixerApplication kept separate —
each commit builds and passes
make cs+make phpstanon its own:Processinterface andProcessBase— splits the worker intoa process-creation-agnostic half (the TCP/NDJSON connection + timeout timer)
and the creation half. Existing impl renamed to
SpawnedProcess.WorkerRunner— everythingWorkerCommanddoes after the boot,made reusable without a re-boot.
pcntl_fork()parallel worker path —ForkParallelChecker+ForkedProcess, wired intoParallelAnalyser.ProcessPromiseinterface — same split for the Pro worker;existing impl renamed to
SpawnedProcessPromise.FixerWorkerRunner— the post-boot half ofFixerWorkerCommand.pcntl_fork()PHPStan Pro worker path —ForkedProcessPromise,wired into
FixerApplication.Gating
Opt-in and conservative —
ForkParallelChecker::isSupported()requires allof:
PHPSTAN_PARALLEL_FORK=1in the environment;ext-pcntl+ext-posixfunctions available;concurrently from forked children and doing so corrupts analysis results.
When not supported, the existing spawn path is used unchanged.
Status / caveats
Experimental. With the gate above, the
workerfork path produces resultsidentical to the spawn path and runs noticeably faster on PHPStan's own source.
The Pro (
fixer:worker) fork path is wired symmetrically but could only besmoke-tested here — the full Pro flow needs the Pro PHAR. There is also a
separate, still-unexplained result discrepancy between fork and spawn under
opcache.enable_cli=0that wants more investigation before this is anythingmore than an experiment.
🤖 Generated with Claude Code