Editing Openai/693c0f4f-255c-8008-92e9-0cd44c6d6226 (section)

==== Even though it’s not a drop-in replacement for dataset-conditioning, there are interesting uses: ====

===== - Train an encoder E: LoRA → δ (basically Delta Activations). =====
* Train a decoder G: δ → LoRA.
* Now: - Edit δ (add, interpolate, project, do PCA or whatever), - Decode back to LoRA.

This gives you:
* A smooth latent editing space for LoRAs, like a StyleGAN latent space but for adapters.
* Strong prior: G only outputs LoRAs similar to those seen during training, reducing weird failure LoRAs.

Weakness:
* It’s still limited to the convex-ish hull of behaviors in your LoRA zoo.
* Doesn’t help you create fundamentally new capabilities you’ve never finetuned for.

===== If δ behaves compositional-ish (e.g., “math + code ≈ math + code in delta space”), you can: =====
* Use δ-space as the slot space for task factors.
* Then G(δ) is the generator that instantiates those factors as a concrete LoRA.

This connects nicely to the compositional paper you read earlier:
* δ plays the role of “slots” (task factors),
* G is the generator f,
* LoRA weights are “images” in that manifold.

You can then play generative games like:
* δ_new = δ_A (domain) + δ_B (behavior style)
* LoRA_new = G(δ_new)

Again the limitation: you’re composing known high-level behaviors; you still don’t get a new capability without some supervision somewhere.

===== A more powerful approach would be: =====
* Condition G on both δ and dataset / prompts, e.g. condition=(δprior,prompt stats)\text{condition} = (\delta_{\text{prior}}, \text{prompt stats})condition=(δprior,prompt stats)
* δ_prior: your best guess of “what kind of task this is like” (e.g. from retrieval in Δ-space).
* Prompt stats: information from the new dataset, like DnD.

Then you get:
* Prior over LoRA space from δ (reuse of existing skills),
* Fine-grained, task-specific signal from prompts.

This could make adaptation faster and more stable than using prompts alone.