Recent easy corruption of nodes on latest/nightly/testing tag for sync & bucket corruption log for fresh sync #911
Description
What version are you using?
multiple of
stellar/quickstart:nightly
stellar/quickstart:latest
stellar/quickstart:testing
at the moment being "Id": "sha256:e7f05fac9ce971eccaabfd528297380c682f39f0d687c02bc7547d273a360145",
What did you do?
nodes broken , reworked multiple times with different tags
What did you expect to see?
working
What did you see instead?
always have postgres broken
2026-03-02T06:18:44.085 [default INFO] Assigning calculated value of 2 to FAILURE_SAFETY
current exception: std::runtime_error("Bucket directory is corrupt")
backtrace:
0: stellar::printCurrentBacktrace()
at /wd/src/util/Backtrace.cpp:27:22
1: stellar::printBacktraceAndAbort()
at /wd/src/main/main.cpp:140:5
2: <unknown>
3: <unknown>
4: __cxa_throw
5: loadLastKnownLedgerInternal
at /wd/src/ledger/LedgerManagerImpl.cpp:550:9
6: start
at /wd/src/main/ApplicationImpl.cpp:805:21
7: runApp
at /wd/src/main/ApplicationUtils.cpp:117:10
8: operator()
at /wd/src/main/CommandLine.cpp:1638:20
__invoke<(lambda at main/CommandLine.cpp:1576:9) &>
at /usr/lib/llvm-20/bin/../include/c++/v1/__type_traits/invoke.h:179:25
__call<(lambda at main/CommandLine.cpp:1576:9) &>
at /usr/lib/llvm-20/bin/../include/c++/v1/__type_traits/invoke.h:243:12
__invoke_r<int, (lambda at main/CommandLine.cpp:1576:9) &>
at /usr/lib/llvm-20/bin/../include/c++/v1/__type_traits/invoke.h:273:10
operator()
at /usr/lib/llvm-20/bin/../include/c++/v1/__functional/function.h:167:12
operator()
at /usr/lib/llvm-20/bin/../include/c++/v1/__functional/function.h:319:10
9: operator()
at /usr/lib/llvm-20/bin/../include/c++/v1/__functional/function.h:436:12
operator()
at /usr/lib/llvm-20/bin/../include/c++/v1/__functional/function.h:995:10
runWithHelp
at /wd/src/main/CommandLine.cpp:479:12
10: run
at /wd/src/main/CommandLine.cpp:1570:12
2026-03-02T06:14:18.407 [default INFO] Assigning calculated value of 2 to FAILURE_SAFETY
current exception: std::runtime_error("sqlite3_statement_backend::loadOne: database disk image is malformed")
2026-03-02 06:15:19 UTC [120668-1607] stellar@horizon ERROR: could not read block 31756 in file "base/16384/16401": read only 0 of 8192 bytes
2026-03-02 06:15:19 UTC [120668-1608] stellar@horizon STATEMENT: SELECT COALESCE(MAX(sequence), 0) FROM history_ledgers
even re-sync with diff image tag still seeing this