Editing Openai/69191dd5-6360-800d-b47b-7ebcc8b69dc8 (section)

===== - Meta reports Llama-3.1 405B was trained on 15T+ tokens, using 39.3M H100-80GB GPU-hours (700 W TDP class). Meta AI<ref>{{cite web|title=Meta AI|url=https://ai.meta.com/blog/meta-llama-3-1/|publisher=Meta AI|access-date=2025-11-16}}</ref> =====
* Architecture: vanilla giant decoder-only Transformer (no MoE), extended context, etc. datacamp.com<ref>{{cite web|title=datacamp.com|url=https://www.datacamp.com/blog/llama-3-1-405b-meta-ai|publisher=datacamp.com|access-date=2025-11-16}}</ref>
* Rubin VR-200 is Nvidia’s next-gen data-center training GPU in the Rubin family: - Positioned as the workhorse training GPU alongside Rubin CPX (long-context prefill) and R100 decode parts. Futurum<ref>{{cite web|title=Futurum|url=https://futurumgroup.com/insights/nvidias-new-rubin-cpx-targets-future-of-large-scale-inference/|publisher=futurumgroup.com|access-date=2025-11-16}}</ref> - VR-200-based platforms (e.g. NVL144 / Rubin + Vera racks) are targeted at multi-exaflop FP4/FP8 training/inference with extreme rack-level power (approaching 1 MW racks). NVIDIA Newsroom<ref>{{cite web|title=NVIDIA Newsroom|url=https://nvidianews.nvidia.com/news/nvidia-unveils-rubin-cpx-a-new-class-of-gpu-designed-for-massive-context-inference|publisher=NVIDIA Newsroom|access-date=2025-11-16}}</ref>
* Exact per-GPU TFLOPs for VR-200 aren’t public yet; safe assumption is significantly above Blackwell GB200 and well above H100.

So the “standard Llama-3 400B on Rubin” scenario is:

: