Skip to content

fix(breakpad): pre-load psapi to prevent frequent loader deadlock#1502

Closed
jpnurmi wants to merge 2 commits intomasterfrom
jpnurmi/ci/breakpad-dump-inflight
Closed

fix(breakpad): pre-load psapi to prevent frequent loader deadlock#1502
jpnurmi wants to merge 2 commits intomasterfrom
jpnurmi/ci/breakpad-dump-inflight

Conversation

@jpnurmi
Copy link
Collaborator

@jpnurmi jpnurmi commented Feb 2, 2026

Closes #1501

Summary

Fixes a deadlock that occurred when capturing crash dumps on Windows while network requests were in flight. See #1501 for detailed stack traces and investigation.

Problem

When a crash occurs, Breakpad calls MiniDumpWriteDump which internally loads psapi.dll at runtime for module enumeration. If another thread (e.g., WinHTTP worker) has pending work in the parallel DLL loader, this deadlocks in LdrpDrainWorkQueue.

Solution

Pre-load psapi.dll during Breakpad backend startup, ensuring it's already loaded before any crash handling occurs.

Test plan

  • Verified fix locally on Windows Server 2022
  • CI passes on all platforms

MiniDumpWriteDump calls LoadPsApi at runtime to enumerate modules.
If this happens while the parallel DLL loader has pending work,
it can deadlock in LdrpDrainWorkQueue.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@jpnurmi jpnurmi force-pushed the jpnurmi/ci/breakpad-dump-inflight branch from f1c31f3 to 87e3c58 Compare February 3, 2026 09:24
@jpnurmi jpnurmi changed the title ci: debug Windows 32-bit hang fix(breakpad): pre-load psapi to prevent frequent loader deadlock Feb 3, 2026
@jpnurmi jpnurmi force-pushed the jpnurmi/ci/breakpad-dump-inflight branch from 6044e10 to e59f32a Compare February 3, 2026 10:06
@jpnurmi
Copy link
Collaborator Author

jpnurmi commented Feb 3, 2026

Another deadlock occurred in the same test with this workaround applied...

[2964] Main thread

>	ntdll.dll!_NtWaitForSingleObject@12()	Unknown
 	KernelBase.dll!WaitForSingleObjectEx()	Unknown
 	KernelBase.dll!_WaitForSingleObject@8()	Unknown
 	sentry.dll!google_breakpad::ExceptionHandler::WriteMinidumpOnHandlerThread(_EXCEPTION_POINTERS * exinfo, MDRawAssertionInfo * assertion) Line 730	C++
 	sentry.dll!google_breakpad::ExceptionHandler::HandleException(_EXCEPTION_POINTERS * exinfo) Line 514	C++
 	KernelBase.dll!_UnhandledExceptionFilter@4()	Unknown
 	ntdll.dll!___RtlUserThreadStart@8()	Unknown
 	ntdll.dll!@_EH4_CallFilterFunc@8()	Unknown
 	ntdll.dll!__except_handler4_common()	Unknown
 	ntdll.dll!__except_handler4()	Unknown
 	ntdll.dll!ExecuteHandler2@20()	Unknown
 	ntdll.dll!ExecuteHandler@20()	Unknown
 	ntdll.dll!_KiUserExceptionDispatcher@8()	Unknown
 	vcruntime140d.dll!748d8425()	Unknown
 	[Frames below may be incorrect and/or missing, no symbols loaded for vcruntime140d.dll]	
 	sentry_example.exe!trigger_crash(...) Line 285	C
 	sentry_example.exe!_mainCRTStartup()	Unknown
 	cccccccc()	Unknown
 	kernel32.dll!@BaseThreadInitThunk@12()	Unknown
 	ntdll.dll!___RtlUserThreadStart@8()	Unknown
 	ntdll.dll!__RtlUserThreadStart@8()	Unknown
  • [4872] sentry-http
>	ntdll.dll!_NtWaitForAlertByThreadId@8()	Unknown
 	ntdll.dll!_RtlSleepConditionVariableCS@12()	Unknown
 	KernelBase.dll!_SleepConditionVariableCS@12()	Unknown
 	sentry.dll!worker_thread(void * data) Line 258	C
 	kernel32.dll!@BaseThreadInitThunk@12()	Unknown
 	ntdll.dll!___RtlUserThreadStart@8()	Unknown
 	ntdll.dll!__RtlUserThreadStart@8()	Unknown

[6120] ntdll TppWorkerThread

>	ntdll.dll!_NtWaitForSingleObject@12()	Unknown
 	ntdll.dll!LdrpDrainWorkQueue()	Unknown
 	ntdll.dll!_LdrShutdownThread@0()	Unknown
 	ntdll.dll!RtlExitUserThread()	Unknown
 	ntdll.dll!_TppWorkerThread@4()	Unknown
 	kernel32.dll!@BaseThreadInitThunk@12()	Unknown
 	ntdll.dll!___RtlUserThreadStart@8()	Unknown
 	ntdll.dll!__RtlUserThreadStart@8()	Unknown

2 threads

>	ntdll.dll!_NtWaitForSingleObject@12()	Unknown
 	ntdll.dll!LdrpDrainWorkQueue()	Unknown
 	ntdll.dll!_LdrpInitializeThread@4()	Unknown
 	ntdll.dll!__LdrpInitialize@8()	Unknown
 	ntdll.dll!_LdrpInitializeInternal@8()	Unknown
 	ntdll.dll!_LdrpInitialize@8()	Unknown
 	ntdll.dll!LdrInitializeThunk()	Unknown

@jpnurmi
Copy link
Collaborator Author

jpnurmi commented Feb 3, 2026

Not convinced this helps much. Pre-loading psapi.dll only helps with the specific deadlock that was easiest to reproduce.

@jpnurmi jpnurmi closed this Feb 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: Windows (Old VS, 32-bit) hangs

1 participant