Explore and implement distributed LLM

[jeremymanning]

The attached whitepaper describes an idea for a distributed text diffusion-based LLM

parallel_mesh_of_diffusers_whitepaper.pdf

There are several uses:

1. the LLM is, in and of itself, a resource: it represents a continually growing and improving language model that is free to everyone and remains at the cutting edge
2. the LLM guides developments and improvements of the world-compute network itself
3. the LLM carries out regular security audits and spot checks of jobs and use management of the broader network

Because the LLM is fundamentally mesh-based, it operates at fractal scales depending on needs in the moment:

• Max intelligence uses ALL available nodes to construct one "super" LLM: if scaled to all compute on earth, this would represent by far the largest and most powerful language model in existence. It could potentially make enormous advances for human society.
• At intermediate scales, resources are drawn upon according to need; more important, larger, and more complex problems are allocated larger pools of nodes
• At the smallest scales, for very basic low-level tasks, even a single node with modest hardware could operate as a simple model

Given the whitepaper and the above clarifications: explore and implement this model

[jeremymanning]

Scaffolding in place; core inference deferred — not closing.

Spec 004 (PR #59) shipped the mesh LLM orchestration shell:

• Router that selects K-of-N experts by throughput (src/agent/mesh_llm/router.rs)
• Expert registry + ExpertNode with LLaMA-3 tokenizer invariant (src/agent/mesh_llm/expert.rs)
• Sparse logit aggregator with weighted averaging + temperature sampling (src/agent/mesh_llm/aggregator.rs)
• Safety tier classifier + governance kill switch (src/agent/mesh_llm/safety.rs)
• Self-prompting loop scaffolding (src/agent/mesh_llm/self_prompt.rs)
• Agent subsetting for parallel domain streams (src/agent/mesh_llm/subset.rs)
• gRPC service handlers (src/agent/mesh_llm/service.rs)
• ~1,697 lines of Rust + 5 integration tests passing

What is NOT yet implemented:

• src/agent/mesh_llm/expert.rs load_model() is a placeholder — no actual candle_transformers::models::llama::Llama::load() call. Comment in source explicitly says "in production this would use...".
• No expert node actually loads LLaMA-3-8B weights or produces real logits.
• service.rs self-describes as stub — 'no real inference yet'.
• The aggregator has correct math but has only been exercised with synthetic logit vectors, never with real model outputs.

The scaffolding is protocol-correct: once load_model() is wired to real candle inference and experts can produce genuine top-256 logits from a running LLaMA-3 quantized model on a GPU, the rest of the pipeline works. But that's the core thing this issue asked for, and it has not been built. Leaving open for a future spec.

[jeremymanning]

Spec 005 progress — PR #61

STATUS: Deferred per user instruction. Architectural direction pinned; implementation not started.

Spec 005 (PR #61) pins the target architecture in specs/005-production-readiness/spec.md US6 (FR-022–FR-029 + FR-028a), per the whitepaper notes/parallel_mesh_of_diffusers_whitepaper.pdf:

• Distributed masked-discrete-diffusion (NOT autoregressive transformer ensembling — explicitly ruled out)
• Dream-class 7B backbone (LLaDA 8B primary; Dream 7B / DiffuLLaMA fallbacks)
• SSD-2-style specialized experts contributing per-denoising-step score signals
• PCG (Predictor-Corrector Guidance) score composition, NOT uniform mean-averaging (per Razafindralambo et al. TMLR 2026)
• ParaDiGMS parallel denoising via Picard iteration (≥ 2× wall-clock speedup target)
• DistriFusion stale-activation pipelining over libp2p (≥ 50 % WAN RTT masked behind compute target)
• Petals-style sharded hosting over libp2p with DCUtR hole-punching
• Target smoke hardware: 3 GPUs tensor01 + 3 GPUs tensor02 + tc netem for controlled WAN benchmark

Implementation scope (21 tasks T066–T086 in specs/005-production-readiness/tasks.md):

• Remove existing AR-ensemble src/agent/mesh_llm/ module
• New src/agent/mesh_llm_diffusion/{backbone,expert,pcg,paradigms,distrifusion,scheduler,safety,service}.rs
• New proto/mesh_llm_diffusion.proto (spec already written in contracts/grpc-mesh-llm-diffusion.md)
• LLaDA-8B weights downloaded + CID-mirrored
• 6-GPU cross-machine smoke test

Risk-flagged as HIGH in tasks.md Risk Flag section — no prior end-to-end system combining distributed diffusion + Petals-libp2p hosting exists in published research. Mitigations include splitting across contributors + early benchmark infrastructure + timeboxing + fallback model.

Remains entirely open; deferred from spec 005 per user decision to tackle in a dedicated session.

See notes/session-2026-04-19-spec-005-kickoff.md.