Editing Openai/693c0f4f-255c-8008-92e9-0cd44c6d6226 (section)

==== 6. Risks / Weaknesses (the honest part) ====
* δ is inherently lossy. Even with a good probe set, different LoRAs can share similar Delta Activations. So δ can’t fully disambiguate adapters; improvements might be modest rather than dramatic.
* Probe bias. Fixed generic probes might underspecify behavior for narrow domains (e.g., some specialized coding style), and δ could mislead the generator.
* Compute overhead. Computing exact δ for generated LoRAs inside the training loop doubles forward passes for probe prompts. You’ll probably need an approximate δ-head HHH, which adds its own approximation error.
* Still no theory. Unlike the compositional generator-identifiability work, there is no clean theorem saying “this δ-regularized generator must produce the right LoRA.” This will be an empirical improvement story, not a theory paper.

But as an incremental, concrete improvement to data-to-LoRA, this is very defensible: it piggybacks on two existing ideas (DnD + Delta Activations), adds a new, plausible type of supervision, and gives lots of measurable knobs to experiment with.

If you want, next step we can narrow down to one main variant (probably multi-task A or two-stage B), and I can help you turn this into a 1–2 page “mini-proposal” with more formal notation and maybe pseudo-code.