thaw

thaw is the fork() primitive for live LLM inference. snapshot a running session — weights, KV cache, and scheduler state — into a portable file, then fork N children that skip the cold prefill and diverge from the fork point. on an H100, a pre-warmed pool forks at 0.88s median per round versus a ~34... Read more