Editing Openai/692db884-c2d8-800c-a357-aa2a0e684fbe (section)

==== Compression sounds like extra work, but here’s why it can help speed overall: ====
* Less data shoved around A huge amount of time on GPUs goes into: - Moving data from VRAM → SMs (cores) - Moving data between buffers, caches, layers If the hook is storing compressed representations: - There are simply fewer bytes to move for many operations. - IO-bound parts of the workload can get faster.
* Decode is massively parallel The decompression logic runs in a way that: - Exploits the GPU’s parallelism. - Turns “extra work” into something the hardware is very good at. So even though you “add” work (decode), you often subtract more work in reduced memory traffic.
* Less offload thrash If you needed CPU/NVMe offload before: - You were paying PCIe latency and disk/DRAM bandwidth costs. - The hook lets more of the working set live in VRAM, so: - Fewer device ↔ host transfers - Less paging, fewer stalls

Overall effect on speed:
* For memory-bound or IO-heavy workloads: 👉 Often faster and/or can handle more tokens per second.
* For purely compute-bound workloads (where VRAM wasn’t a limit): 👉 Usually neutral or a small overhead, since the benefit is mostly about capacity and IO, not raw flops.

So it’s like this:

: