Editing Openai/6939b23e-87cc-8001-bdd9-9546d1143ce9 (section)

=== This part is crucial. ===

Between token generation steps:
* The model does not keep internal state
* The model does not remember what it was planning
* The model does not store a chain of thought

Everything is reconstructed from the text prompt + whatever tokens were already generated.

Internally it works like this:
# Take the entire conversation text (prompt + generated output so far)
# Feed it into the transformer
# Predict the next token
# Output the token
# Throw away all internal activations
# Begin step 1 again with the new text

So even during a single answer, the model is not running continuously.
It is re-running the whole network again and again.