Editing Openai/6936bc88-f39c-800d-8af0-70d83304b81c (section)

=== 1️⃣ Kernels most ripe for auto-tuned improvement ===

===== ✔ Fused MLP kernels (GEMM + RMSNorm + residual + activation fusion) =====

Typical hand-tuned fusion under-performs on:
* odd sizes
* MoE batches
* small matmuls
* variable sequence lengths
* multi-query patterns

Auto-tuning gain: 15–35% on the MLP subsystem.

===== ✔ MoE router + token dispatcher kernels =====

This is currently a mess everywhere — high register pressure, low occupancy, tons of wasted bandwidth.

Auto-tuning gain: 30–70% depending on model density.

For DeepSeek / Mixtral / Qwen MoE, this is huge.

===== ✔ KV-cache read/write, fused rotary, fused QK, fused KV update =====

The KV system is horribly inefficient on all GPUs today.

Auto-tuning gain: 20–50% on the KV path.

With long context (≥128k), this becomes the #1 bottleneck.

===== ✔ Small and medium GEMMs (the silent enemy) =====

Most models run thousands of 128×X and 256×X GEMMs per token.
Human-optimized CUDA/ROCm is bad at these.

Auto-tuning can:
* select better tiles
* restructure loops
* rebalance wave occupancy

Auto-tuning gain: 10–40% depending on shapes.