Editing Openai/693c0f4f-255c-8008-92e9-0cd44c6d6226 (section)

===== Intuitively it’s tempting to think: “but d≪Nd \ll Nd≪N, so the representation can’t just memorize!” =====
Unfortunately, it absolutely can.

A deep net encoder with small output dimension can still:
* map each training image xix_ixi to some distinct vector zi∈Rdz_i \in \mathbb{R}^dzi∈Rd,
* and a classifier head can memorize a mapping {zi}→{1,…,N}\{z_i\} \to \{1,\dots,N\}{zi}→{1,…,N}.

Key points:
# Continuous codes carry lots of information. Even a 32-dim float vector can encode absurdly many distinct codes. We are not limited to “one dimension per class” – that’s only true if you force something like one-hot or linear separability constraints and low numeric precision.
# Memorization in low-dim is easy for finite data. For a finite set of NNN points, you can always construct an injective map xi↦zi∈Rdx_i \mapsto z_i \in \mathbb{R}^dxi↦zi∈Rd for very small ddd (even d=2d=2d=2 for reasonable NNN). Then the classifier head (linear or shallow MLP) just learns decision regions around those codes. Geometrically: you only need each training point to sit in a little “cell” of representation space that maps to its label. No requirement those cells are arranged in a way that reflects compositional structure.
# No pressure to share factors. The bottleneck doesn’t force the encoder to separate “animal slot” vs “background slot”; it just forces some kind of compression. The easiest compression for SGD might be something like “two big clusters corresponding to (camel,desert) images vs (penguin,snow) images”, not “4 independent factors”.

So the encoder can happily learn a highly entangled representation that just supports “memorize N instance labels” with no notion of compositionality.