Editing Openai/692db884-c2d8-800c-a357-aa2a0e684fbe (section)

==== LoreTokens change the game in ways a generic copycat can’t trivially match: ====

===== Without LoreTokens, they can only compress what they see numerically: =====
* “Oh, lots of zeros here.”
* “Oh, these values sort of cluster.”
* “Oh, this row looks like that row.”

That’s fine, but:
* It’s bounded by whatever statistical patterns happen to exist in that tensor.
* They have to treat everything as “anonymous floats.”

With LoreTokens:
* You’re saying: >
* So you can encode that as compact symbols and tiny deltas instead of full raw tensors.

Result: higher compression ratios and more stable performance, especially across repeated schemas.

===== LoreTokens basically say: =====
* “This structure shows up everywhere, so I’ll store it once as a schema and just reference it symbolically.”

That gives you:
* Huge wins on repetition (same indicator layout, same DB row structure, same meta pattern).
* Minimal overhead per additional instance.

A clone without LoreTokens:
* Has to “rediscover” every pattern statistically, every time.
* Can’t reuse high-level structure across runs, datasets, or components in a principled way.

So their effective compression ceiling is lower, and more workload-dependent.

===== With semantic structure, you can: =====
* Decode only the relevant part of a compressed block because you know exactly what each symbol corresponds to.
* Skip reconstructing chunks that won’t be touched in this operation.

That means:
* Less decode work
* Less temporary VRAM
* Better cache locality

A generic compressor, by contrast, usually has to:

: 

So again: more compute + more VRAM used per operation → weaker practical gain.