Editing Openai/691c33ba-8898-800c-b30f-1383bae461b1 (section)

===== 1.2 Use half precision end-to-end (not just autocast) =====

Impact: medium–high • Complexity: low

Right now your decorator does:

<syntaxhighlight lang="python">self.autocast = torch.amp.autocast('cuda')

</syntaxhighlight>

Consider:
* Put models permanently in FP16: ``<code>python model = model.half().eval().to("cuda") <syntaxhighlight>- Make sure inputs are float16 too, or wrapped in autocast only around the actual forward.

For many transformer-ish models (RFDetr, DINO) this is a solid speed + memory win. Watch out for:
* Any ops that don’t support FP16 (might need a small to(torch.float32) around them).

===== 1.3 Make your decorator more focused =====

Impact: small–medium • Complexity: low

Right now torch_inference_mode_block() is wrapping everything under inference + autocast (including Python control logic). Nice and clean, but you can:
* Narrow the scope so it only wraps the model forward passes, not the whole big function that does pre/post-processing, SAHI tiling, etc.
* That reduces some context-manager overhead and makes it easier to ensure that: - SAHI / CPU side runs in normal mode - Only GPU forwards are in autocast/inference_mode.

Not huge, but free.

===== 1.4 Turn on channels_last + pinned memory =====

Impact: medium • Complexity: low

For conv-heavy parts (RTMPose especially):

</syntaxhighlight>python
model = model.to(memory_format=torch.channels_last)
input = input.to(device, memory_format=torch.channels_last, non_blocking=True)

<syntaxhighlight>
And for DataLoader / frame feeding:
* Use pin_memory=True and .to(device, non_blocking=True) to overlap H2D copies with compute.

===== 1.5 Reduce resolution / SAHI aggressiveness =====

Impact: huge if you can afford it • Complexity: low
* Check if you can: - Run RFDetr on downscaled frames (e.g. 1080p → 720p), then rescale boxes back. - Make SAHI tiles larger and with less overlap: - Fewer tiles → fewer forwards.
* Often, a small drop in detection quality = massive speedup.

===== 1.6 cuDNN / PyTorch 2 compiler =====

Impact: small–medium per model, sometimes big • Complexity: low
* Set: </syntaxhighlight>python torch.backends.cudnn.benchmark = True  # if varying input sizes <syntaxhighlight>- If you’re on PyTorch 2: </syntaxhighlight>python model = torch.compile(model, mode="max-autotune") </code>`` Especially for RTMPose and DINO; watch for weird dynamic shapes (try to keep SAHI tile sizes fixed).