feat(zero-cache): 2x faster initial sync#5704
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Bencher Report| Branch | mlaw/faster-initial |
| Testbed | Linux |
Click to view all benchmark results
| Benchmark | File Size | Benchmark Result kilobytes (KB) (Result Δ%) | Upper Boundary kilobytes (KB) (Limit %) |
|---|---|---|---|
| zero-package.tgz | 📈 view plot 🚷 view threshold | 1,963.09 KB(+0.74%)Baseline: 1,948.69 KB | 1,987.67 KB (98.76%) |
| zero.js | 📈 view plot 🚷 view threshold | 269.36 KB(-0.02%)Baseline: 269.40 KB | 274.79 KB (98.02%) |
| zero.js.br | 📈 view plot 🚷 view threshold | 71.45 KB(+0.06%)Baseline: 71.41 KB | 72.84 KB (98.10%) |
tantaman
changed the title
|
| Branch | mlaw/faster-initial |
| Testbed | self-hosted-metal |
Click to view all benchmark results
| Benchmark | Throughput | Benchmark Result operations / second (ops/s) (Result Δ%) | Lower Boundary operations / second (ops/s) (Limit %) |
|---|---|---|---|
| src/db/pg-copy.bench.ts > pg-copy benchmark > copy | 📈 view plot 🚷 view threshold | 24.76 ops/s(+4.93%)Baseline: 23.60 ops/s | 22.80 ops/s (92.11%) |
|
| Branch | mlaw/faster-initial |
| Testbed | self-hosted-metal |
Click to view all benchmark results
| Benchmark | Throughput | Benchmark Result operations / second (ops/s) x 1e3 (Result Δ%) | Lower Boundary operations / second (ops/s) x 1e3 (Limit %) |
|---|---|---|---|
| src/client/custom.bench.ts > big schema | 📈 view plot 🚷 view threshold | 117.27 ops/s x 1e3(+7.84%)Baseline: 108.75 ops/s x 1e3 | 95.86 ops/s x 1e3 (81.74%) |
| src/client/zero.bench.ts > basics > All 1000 rows x 10 columns (numbers) | 📈 view plot 🚷 view threshold | 1.51 ops/s x 1e3(+0.03%)Baseline: 1.51 ops/s x 1e3 | 1.06 ops/s x 1e3 (69.93%) |
| src/client/zero.bench.ts > pk compare > pk = N | 📈 view plot 🚷 view threshold | 37.82 ops/s x 1e3(-0.06%)Baseline: 37.85 ops/s x 1e3 | 36.26 ops/s x 1e3 (95.87%) |
| src/client/zero.bench.ts > with filter > Lower rows 500 x 10 columns (numbers) | 📈 view plot 🚷 view threshold | 1.89 ops/s x 1e3(-8.24%)Baseline: 2.06 ops/s x 1e3 | 1.70 ops/s x 1e3 (89.75%) |
|
| Branch | mlaw/faster-initial |
| Testbed | self-hosted-metal |
⚠️ WARNING: Truncated view!The full continuous benchmarking report exceeds the maximum length allowed on this platform.
🚨 1 Alert
🐰 View full continuous benchmarking report in Bencher
tantaman
force-pushed
the
mlaw/faster-initial
branch
from
7575539 to
00e7625
Compare
tantaman
force-pushed
the
mlaw/faster-initial
branch
from
00e7625 to
6e0de1a
Compare
@darkgnotic - My theory is that it is faster for PG to get the data out via the binary format (https://github.com/postgres/postgres/blob/f8eec1ced6979157cbf517b5bbf617b82a01a397/src/backend/commands/copyto.c#L1421-L1450). IIUC, text copy has to scan every byte coming out to do proper escaping. Binary copy allows PG to just dump the raw bytes to the socket. This would mean that it wasn't the network that was the problem but the time to get bytes out of PG, which would look like the network to an external consumer. WDYT? I think this also makes sense since gigabugs is only 1GB so it should transfer within 1-2 seconds, not the 10-20 seconds we see it take. Measuring the data over the wire, the binary format is actually slightly larger for gigabugs than text. 1,000 MB vs 967 MB |
tantaman
force-pushed
the
mlaw/faster-initial
branch
from
1e4e223 to
6505d06
Compare
Ah, this would make sense. The escaping probably costs us a bit of time on the parsing side too.
Yeah, that is what dissuaded me from pursuing this option way back in the day. But I didn't do the due diligence of actually benchmarking it end-to-end. Thank you! |
Swaps to the binary protocol for copying from PG rather than using the text protocol. ``` ┌──────────────┬──────────────────┬───────────────────────────┬─────────┐ │ │ Main (text COPY) │ This branch (binary COPY) │ Speedup │ ├──────────────┼──────────────────┼───────────────────────────┼─────────┤ │ Total time │ 19.45s │ 9.86s │ 1.97x │ ├──────────────┼──────────────────┼───────────────────────────┼─────────┤ │ Rate │ 142.0K rows/s │ 280.0K rows/s │ 1.97x │ ├──────────────┼──────────────────┼───────────────────────────┼─────────┤ │ Replica size │ 1256.9 MB │ 1256.9 MB │ same │ └──────────────┴──────────────────┴───────────────────────────┴─────────┘ ```
2 participants
Swaps to the binary protocol for copying from PG rather than using the text protocol.