Breaking the Monolith: Prefill/Decode Disaggregation and KV Cache Tiering in Modern LLM Serving
Why running LLM inference on homogeneous GPU pools hits a wall—and how separating compute from memory bandwidth across HBM, DRAM, and NVMe unlocks massive goodput.
DevOps & AI Architecture Dispatch
Join engineers receiving hands-on writeups on Kubernetes clustering, Terraform GitOps, distributed ML infrastructure, and low-latency LLMOps. No marketing fluff—strictly technical notes.
Join the Discussion on Breaking the Monolith: Prefill/Decode Disaggregation and KV Cache Tiering in Modern LLM Serving
Comments and Q&A are powered by GitHub Discussions. To enable them with zero maintenance, add your repository IDs to your environment variables or visit giscus.app.
Running a 552B MoE on Two Desktop Workstations: Dissecting MiaAI’s DeepSeek V4.1 Flash EXL3 Stack
How fine-grained 2.9 bpw mixed quantization, unified memory management, and DSpark speculative decoding fit DeepSeek V4.1 Flash across dual NVIDIA DGX Sparks.
- #AI Infrastructure
- #AI / ML
- #MLOps
Squeezing a 27B LLM into a $700 GPU: A Masterclass in AI Platform Engineering
How low-level inference engineering, 4-bit quantization, and speculative decoding push 1,000+ tokens/sec out of a consumer RTX 3090.
- #AI / ML
- #AI Infrastructure
- #GenAI
The Sub-$1,000 MoE Cluster: Serving a 70B Mixture-of-Experts with Speculative Expert Paging
How asynchronous NVMe streaming, router logit lookahead, and tiered ring buffers push production-grade inference out of commodity hardware.
- #AI Infrastructure
- #AI / ML