AI Safety Beyond the Model: Like Hospital Access Control Systems

For the past two years, when people discuss AI safety, they always circle back to which model to trust. Closed-source, open-source, domestic, overseas—arguments often devolve into model parameters and leaderboards. But I increasingly feel this question is off-target.

For AI truly entering enterprise processes, the security boundary shouldn't be limited to inside the model. Whether it can read patient information, whether it can call PACS—the API requests and result write-backs are the real life-or-death issues. No matter how human-like the model output is, if permissions aren't managed well, it's just a tool that might run away on its own.

Recently, I saw discussions by Anthropic regarding zero-trust for AI Agents. The statement is very direct: Agents entering production environments must be constrained by identity, permissions, tool gateways, and audit systems, not treated merely as large models connected to tools.

This rings particularly true in medical AI. For imaging AI deployment, beyond model scores, there are many hard problems. Has clinical validation been done? Is doctor feedback acceptable in actual use? Do results enter the image-reading workflow? Is there a trail? Can it be traced? No matter how beautiful a model looks on a test set, if it makes doctors spend two extra minutes verifying in the department, it will struggle to survive.

So, AI safety moving away from the model indicates people are starting to admit that model self-discipline is unreliable. External control is more reliable. Who allowed it to make calls, what did it call, does it have least privilege, are there logs, can it be stopped with one click if something goes wrong? Enterprises buying AI aren't just buying a system that answers questions; they also need a control layer that is auditable, rollback-capable, and integratable into processes.

Model capability determines whether AI can do the work; permission boundaries determine whether it is qualified to do the work.

AI safety products will increasingly resemble hospital permission, logging, and audit systems. Boring, but valuable.