With Parameter Scale Breaking 2.8 Trillion, How Much Optimization Space Remains for Compilers?

When I saw the news that Kimi K3 topped Frontend Code Arena, my first reaction wasn't excitement but opening a memory bandwidth calculator. With 2.8 trillion parameters, even using FP16 inference, the theoretical memory access for a single forward pass exceeds 5.6TB. At the current H100 bandwidth of 3.35TB/s, pure parameter loading alone takes nearly 2 seconds. This number made me realize immediately that Moonshot AI absolutely cannot be doing dense inference with all parameters activated.