Limits of Local Inference: OpenAI's Mac Update and the Next Battleground for AI Compilers

Here's some data: On an M2 Ultra chip, local inference based on llama.cpp has a time-to-first-token (TTFT) latency of about 120ms, while OpenAI's new Mac app, through an optimized hybrid local+cloud architecture, pushes the TTFT for coding-related requests below 80ms, with memory usage increasing by only 12%. This isn't just simple client wrapping; it's a real-world exercise of AI compilers on edge devices.