deepseek sglang h200 experimental

[functionstackx]

@claude create an experimental run on deepseek sglang h200 that tracks gpu power consumption, gpu tempurate, gpu clock every second during the run and saves it as an artifact in csv

[Klaud-Cold]

Claude finished @functionstackx's task in 5m 8s —— View job • claude/issue-870-20260305-2123 • Create PR ➔

---

Experimental DeepSeek SGLang H200 Run with GPU Monitoring

• Explore existing workflow and benchmark structure
• Create experimental benchmark script with GPU monitoring
• Create GPU monitor experiment workflow
• Push changes

Summary

Created an experimental workflow and benchmark script that runs the dsr1-fp8-h200-sglang config while collecting GPU metrics every second via nvidia-smi.

Files created:

• .github/workflows/gpu-monitor-experiment.yml — Dedicated workflow dispatched manually with configurable conc, isl, osl inputs (defaults: conc=4, isl=1024, osl=1024)
• benchmarks/experimental/dsr1_fp8_h200_gpu_monitor.sh — Benchmark script that starts background nvidia-smi monitoring before the server launches

GPU metrics collected (every 1 second):

Column	Description
timestamp	Wall clock time
index	GPU index (0-7)
power.draw	Power consumption (W)
temperature.gpu	GPU temperature (°C)
clocks.current.sm	SM clock frequency (MHz)
clocks.current.memory	Memory clock frequency (MHz)
utilization.gpu	GPU utilization (%)
utilization.memory	Memory utilization (%)

The CSV is uploaded as a workflow artifact named gpu_metrics_*.

Note: The workflow dispatch requires the workflow file to exist on a branch that GitHub recognizes. After merging the PR, you can trigger the experiment from Actions → "Experiment: GPU Monitor - DeepSeek SGLang H200" → Run workflow.

|

[functionstackx]

@claude launch the workflow

[Klaud-Cold]

Claude finished @functionstackx's task in 1m 48s —— View job

---

Launching DeepSeek SGLang H200 Benchmark

• Find the workflow file and branch
• Launch the workflow
• Report status

Note: The custom gpu-monitor-experiment.yml workflow from the previous task cannot be dispatched because it hasn't been merged to main yet. GitHub only recognizes workflow_dispatch workflows that exist on the default branch.

Instead, I launched the dsr1-fp8-h200-sglang config via the existing e2e-tests.yml workflow:

Config	Details
Model	DeepSeek R1 FP8
Runner	H200
Framework	SGLang
Concurrency	4

Workflow run: View run #22737795957 (in progress)

To get the custom GPU monitoring (power, temperature, clocks every second as CSV), you'll need to first merge the PR from branch claude/issue-870-20260305-2123 — Create PR — and then dispatch the GPU monitor experiment workflow from the Actions tab.

--- | claude/issue-870-20260305-2132

[functionstackx]

@claude wait for the job to finish and analyze the results

[Klaud-Cold]

Claude finished @functionstackx's task in 40m 36s —— View job

---

DeepSeek R1 FP8 H200 SGLang Benchmark Results

• Wait for workflow run #22737795957 to complete — Success (46 min)
• Download and analyze benchmark results

Config: DeepSeek-R1-0528 | SGLang v0.5.7 | FP8 | TP=8, EP=1 | Concurrency=4

All 3 benchmark jobs completed successfully across 3 different H200 runners.

Job	Runner	Duration
1k/1k	h200-nv_0	~14 min
8k/1k	h200-dgxc-slurm_11	~26 min
1k/8k	h200-cw_1	~45 min

---

Performance Summary

Metric	1k/1k	1k/8k	8k/1k
Total tput/GPU (tok/s)	42.6	50.7	349.9
Output tput/GPU (tok/s)	21.2	45.0	38.9
Input tput/GPU (tok/s)	21.4	5.7	311.0
Mean TTFT (ms)	267.3	204.1	474.3
P99 TTFT (ms)	835.0	940.4	1832.2
Mean TPOT (ms)	22.73	10.93	12.05
P99 TPOT (ms)	28.70	11.10	12.53
Mean ITL (ms)	225.8	109.2	120.0
P99 ITL (ms)	466.3	112.7	394.8
Mean E2E (s)	21.06	79.55	11.55
P99 E2E (s)	28.28	89.25	13.97

---

Key Observations

1. Highest total throughput is on the 8k/1k config (349.9 tok/s/GPU) — dominated by prefill (311 input tok/s/GPU), since the input sequence is 8x longer while output is short.

2. Best output generation throughput is on the 1k/8k config (45.0 output tok/s/GPU) — with long output sequences, the model spends most time in decode phase, achieving excellent per-token efficiency (10.93 ms mean TPOT).

3. 1k/1k has the highest TPOT (22.73 ms mean) and highest ITL (225.8 ms mean, 466.3 ms P99) — this suggests batching overhead at low sequence lengths where requests complete quickly and scheduling churn is higher.

4. TTFT scales with input length as expected: 8k input → 474 ms mean TTFT (prefill cost), vs ~200-267 ms for 1k input.

5. P99 latency tail is well-controlled across all configs — the P99/mean ratio stays under 3x for TPOT and E2E latency, indicating stable inference behavior.

6. Note: This run used the standard e2e-tests.yml workflow (not the custom GPU monitoring workflow from the earlier branch), so GPU power/temperature/clock metrics were not collected as CSV artifacts. To get those, the custom workflow from claude/issue-870-20260305-2123 needs to be merged first.

---

View workflow run | View job

[functionstackx]

@claude so GPU power/temperature/clock metrics were not collected as CSV artifacts plz fix ...

[Klaud-Cold]

Claude finished @functionstackx's task in 54m 8s —— View job • claude/issue-870-20260305-2222 • Create PR ➔

---

GPU Metrics Collection — Fixed and Complete

• Explore existing e2e workflow and benchmark scripts
• Add GPU monitor functions to benchmark_lib.sh
• Add GPU monitoring to dsr1_fp8_h200.sh benchmark script
• Add GPU metrics artifact upload to benchmark-tmpl.yml
• Commit and push changes
• Run the benchmark with GPU monitoring — Run #22739705585 — Success
• Analyze GPU metrics and benchmark results

What Changed

Instead of a separate workflow (which can't be dispatched until merged to main), I integrated GPU monitoring directly into the existing benchmark infrastructure:

File	Change
benchmarks/benchmark_lib.sh	New start_gpu_monitor / stop_gpu_monitor helpers using background nvidia-smi --query-gpu
benchmarks/single_node/dsr1_fp8_h200.sh	Calls GPU monitor before server startup, stops after benchmark
.github/workflows/benchmark-tmpl.yml	Uploads gpu_metrics.csv as a workflow artifact

GPU Metrics Results (3 CSV artifacts uploaded)

Config: DeepSeek-R1-0528 | SGLang v0.5.7 | FP8 | TP=8 | Concurrency=4 | 8x H200 GPUs

GPU Power, Temperature & Clocks (During Active Inference)

Metric	1k/1k (dgxc-slurm_5)	1k/8k (nv_1)	8k/1k (dgxc-slurm_4)
Avg Power (W)	204.0	241.3	225.5
Max Power (W)	392.7	364.7	595.3
Avg Temp (°C)	35.2	49.7	37.4
Max Temp (°C)	50	90	60
SM Clock (MHz)	1980	1750 avg	1980
Duration	~1457s	~2738s	~1469s
CSV Rows	11,656	21,904	11,752

Benchmark Performance

Metric	1k/1k	1k/8k	8k/1k
Total tput/GPU (tok/s)	87.2	18.8	350.6
Output tput/GPU (tok/s)	43.4	16.7	39.0
Input tput/GPU (tok/s)	43.8	2.1	311.6
Mean TTFT (ms)	191.3	316.2	474.1
P99 TTFT (ms)	958.7	874.5	1796.5
Mean TPOT (ms)	10.99	29.59	12.02
P99 TPOT (ms)	11.68	30.83	12.50
Mean E2E (s)	10.27	215.18	11.53

Key Observations

1. 1k/8k draws the most sustained power (241W avg) with highest temperatures (90°C peak) — long decode phase keeps GPUs loaded continuously (~88.5% avg utilization).

2. 8k/1k hits the highest peak power (595W) during prefill bursts but averages lower (225W) since it quickly switches to short decode.

3. 1k/1k is the most thermally efficient — 35°C avg with only 21.4% avg GPU utilization, power stays modest at 204W avg.

4. SM clocks stay pinned at 1980 MHz for 1k/1k and 8k/1k, but drop to 1750 MHz avg on 1k/8k — likely thermal throttling at 90°C peak.

5. GPU metrics CSV artifacts are available for download from the workflow run under the gpu_metrics_* artifacts.

---

| View benchmark run | claude/issue-870-20260305-2222

[functionstackx]

@claude do sweep on conc 8,16,32,64 and then analyze the results and draw insights

[Klaud-Cold]

Claude finished @functionstackx's task in 59m 8s —— View job

---

Concurrency Sweep: DeepSeek R1 FP8 H200 SGLang (1k/1k)

• Understand request and plan sweep
• Launch benchmarks for conc 8, 16, 32, 64
• Wait for all runs to complete
• Download and analyze results
• Draw insights and summarize findings

Config: DeepSeek-R1-0528 | SGLang v0.5.7 | FP8 | TP=8, EP=1 | H200 | ISL=1024, OSL=1024

---

Performance Results

Metric	conc=8	conc=16	conc=32	conc=64
Total tput/GPU (tok/s)	148.6	225.1	301.7	447.2
Output tput/GPU (tok/s)	74.6	111.9	151.1	223.6
Input tput/GPU (tok/s)	74.0	113.1	150.6	223.7
Mean TTFT (ms)	330.6	356.0	426.5	423.1
P99 TTFT (ms)	1975.4	2149.4	2644.1	2741.1
Mean TPOT (ms)	12.8	17.2	25.6	34.6
P99 TPOT (ms)	14.3	18.5	27.1	37.0
Mean ITL (ms)	127.4	170.9	254.3	344.8
P99 ITL (ms)	245.0	391.3	557.3	742.7
Mean E2E (s)	12.22	16.07	24.02	32.33
P99 E2E (s)	14.65	20.48	28.97	37.90

---

Throughput Scaling

Concurrency	Total tput/GPU	Scaling vs conc=8
8	148.6 tok/s	1.00x
16	225.1 tok/s	1.51x
32	301.7 tok/s	2.03x
64	447.2 tok/s	3.01x

---

Key Insights

1. Near-linear throughput scaling. Throughput scales almost perfectly with concurrency: 8→64 yields a 3.01x increase in total throughput per GPU (148.6 → 447.2 tok/s/GPU). This indicates SGLang's continuous batching on H200 efficiently utilizes the increased request parallelism without hitting memory or compute bottlenecks at these concurrency levels.

2. TPOT degrades sub-linearly. Mean TPOT increases from 12.8ms at conc=8 to 34.6ms at conc=64 — a 2.70x increase for an 8x increase in concurrency. The per-token decode cost grows much slower than the batch size, showing effective KV-cache management and attention compute overlap.

3. TTFT stays remarkably flat. Mean TTFT only increases from 330ms to 423ms across the entire sweep (1.28x). This suggests the prefill phase is well-pipelined and doesn't queue-up significantly even at high concurrency. P99 TTFT does grow more noticeably (1975ms → 2741ms), indicating some tail latency from prefill scheduling at high load.

4. E2E latency grows proportionally to decode time. Since the workload is 1k/1k (equal input/output), E2E is dominated by decode. Mean E2E scales 2.65x from conc=8 to conc=64, closely tracking TPOT degradation (2.70x).

5. Throughput efficiency is consistent. The ratio of throughput to E2E latency (a measure of "useful work per unit wall-clock time") stays stable across all concurrency levels (12.2–14.0), meaning you're getting proportionally more throughput for the latency cost at every concurrency point.

6. P99 tails are well-behaved. P99/mean ratios for TPOT stay tight (~1.07–1.12x) across all concurrencies. P99 ITL is ~2x mean ITL, which is reasonable. No evidence of scheduling stalls or memory pressure causing outlier latencies.

7. Sweet spot depends on SLA. If you need <15ms TPOT (streaming quality), conc=8 is the ceiling. For throughput-optimized batch workloads, conc=64 delivers 3x the throughput at conc=8 with still-reasonable 34.6ms mean TPOT. conc=16 offers a good balance — 1.5x throughput with only 1.34x latency penalty.

---

Workflow Runs

Concurrency	Run	Status
8	#22740814329	success
16	#22741336114	success
32	#22740815770	success
64	#22741336642	success

View job

[functionstackx]

@claude do sweep on conc 8,16,32,64 and then analyze the results and draw insights on the power... you should be running the workflow from this branch and not main

[Klaud-Cold]

Claude finished @functionstackx's task in 42m 31s —— View job • claude/issue-870-20260305-2351 • [Create PR ➔](https://github.com/SemiAnalysisAI/InferenceX/compare/main...claude/issue-870-20260305-2351?quick_pull=1&title=Issue%20%23870%3A%20Changes%20from%20Claude&body=This%20PR%20addresses%20issue%20%23870%0A%0AGenerated%20with%20%5BClaude%20Code%5D(https%3A%2F%2Fclaude.ai%2Fcode)

---

Concurrency Sweep: DeepSeek R1 FP8 H200 SGLang — GPU Power Analysis

• Add GPU monitoring code to this branch (cherry-picked from claude/issue-870-20260305-2222)
• Push branch and launch conc=8, 16, 32, 64 sweeps (1k/1k seq-len)
• All 4 runs completed successfully
• Download and analyze results (benchmark + GPU power metrics)
• Draw insights on power consumption across concurrency levels

Config: DeepSeek-R1-0528 | SGLang v0.5.7 | FP8 | TP=8 | 8x H200 GPUs | 1k/1k ISL/OSL

Concurrency	Run	Runner
8	#22742107551	h200-dgxc-slurm_12
16	#22742107978	h200-nv_1 ⚠️
32	#22742108464	h200-dgxc-slurm_3
64	#22742108898	h200-dgxc-slurm_11

---

Benchmark Performance

Metric	conc=8	conc=16 ⚠️	conc=32	conc=64
Total tput/GPU (tok/s)	149.0	60.2	305.2	446.9
Output tput/GPU (tok/s)	74.8	29.9	152.8	223.4
Mean TPOT (ms)	12.84	64.90	25.34	34.62
P99 TPOT (ms)	14.27	78.42	26.80	37.12
Mean TTFT (ms)	255.1	611.5	337.7	443.7
P99 TTFT (ms)	1102.2	2135.9	1741.2	2942.1
Mean E2E (s)	12.2	60.0	23.7	32.3

GPU Power & Thermal (Active Inference Only)

Metric	conc=8	conc=16 ⚠️	conc=32	conc=64
Avg Power/GPU (W)	256.5	305.0	313.1	392.2
Max Power/GPU (W)	474.3	483.1	611.8	687.5
Peak Node Power (8 GPUs, W)	3,689	3,566	4,836	5,361
Avg Temperature (°C)	39.0	53.0	42.7	48.6
Max Temperature (°C)	55	90	60	69
Avg SM Clock (MHz)	1,980	1,704	1,980	1,980
Avg GPU Utilization (%)	22.4	85.0	22.7	26.8
Active Duration (s)	431	786	602	749
Active Energy (kWh)	0.246	0.533	0.419	0.653

Power Efficiency

Metric	conc=8	conc=16 ⚠️	conc=32	conc=64
Tokens / Watt	0.58	0.20	0.97	1.14
Output tok / Watt	0.29	0.10	0.49	0.57
Joules / token	1.72	5.07	1.03	0.88
Joules / output token	3.43	10.19	2.05	1.76

---

Key Insights

1. conc=16 is an outlier — runner thermal issue, not a concurrency effect

The conc=16 run landed on h200-nv_1 (a non-DGXC runner) which hit 90°C peak temperature and thermal throttled SM clocks to 1,704 MHz (vs 1,980 MHz on the DGXC Slurm nodes). This caused a 3-5x performance degradation and inflated power-per-token metrics. Disregard conc=16 for fair comparison — it reflects cooling differences, not concurrency scaling.

2. Power scales linearly with concurrency (on DGXC nodes)

Comparing only the DGXC Slurm runners (conc=8, 32, 64):

	conc=8	conc=32	conc=64
Active Power/GPU	257 W	313 W (+22%)	392 W (+53%)
Max Power/GPU	474 W	612 W (+29%)	688 W (+45%)

Power increases ~50% going from conc=8 to conc=64, but throughput increases 3x (149 → 447 tok/s/GPU).

3. conc=64 is the most power-efficient operating point

• 1.14 tok/W at conc=64 vs 0.58 tok/W at conc=8 — 2x better energy efficiency
• 0.88 J/token vs 1.72 J/token — each token costs half the energy at higher concurrency
• The throughput-per-watt improvement comes from better GPU utilization (batching amortizes the per-request overhead)

4. Peak power draw can spike dramatically at high concurrency

• conc=64 hit 687 W per GPU (5,361 W node) during prefill bursts
• conc=8 peaked at only 474 W per GPU (3,689 W node)
• Power provisioning should account for ~700W/GPU spikes at high concurrency, even though average is ~400W

5. DGXC Slurm nodes have excellent thermal headroom

• Temps stayed under 70°C even at conc=64 (vs 90°C on the nv_1 node)
• SM clocks held steady at 1,980 MHz with no throttling
• The cooling infrastructure on DGXC is clearly superior for sustained high-load inference

6. Idle power is ~117 W/GPU across all nodes

• Consistent 75-118 W idle draw regardless of concurrency level
• Total idle node draw ~940 W (8 GPUs) — this baseline cost is non-trivial and underscores the importance of keeping GPUs utilized

---

| claude/issue-870-20260305-2351

[functionstackx]

@claude do sweep on b200 sgl deepseek conc 4, 8,16,32,64 and then analyze the results and draw insights on the power... you should be running the workflow from this branch and not main