Editing Openai/691a41cd-2efc-800c-9eff-de439224a90d (section)

==== Modern LLMs are mostly limited by VRAM and memory bandwidth, not raw FLOPs: ====
* Huge tensors (weights, activations, KV cache)
* Tons of zeros after ReLU
* Sparsity from pruning
* Quantized models with only a small set of unique values
* Batches where inputs are very similar

Normally, the GPU moves all of that around uncompressed.

This project adds a transparent compression layer under your model that:
# Interposes on CUDA memory calls (malloc/free/memcpy & friends).
# Decides, per-tensor, whether it’s worth compressing.
# Stores a compressed representation in GPU memory.
# Decompresses just-in-time when the tensor is actually needed.

The README targets an RTX 3090 and pitches:
* Effective VRAM going from 24GB → roughly 3–5× effective capacity
* Effective bandwidth multiplying similarly, because less data moves for the same work

The core idea: turn LoreToken semantic compression into a live GPU memory system, not just a file format.