Before AI Acts

AI model safety doesn’t begin and end in the weights

Much of the AI safety conversation focuses on alignment: training models to behave well. But a model trained to behave well is not, by itself, a system you can trust to act.

Before an AI system changes an account, updates a record, or executes a tool call, it needs more than a plausible answer. It needs the right authority, sufficient evidence, checks against current conditions, and an estimate of how reliable its output is. A model may receive policies and approval history in its context, but interpreting that information is not the same as having authorization to act. Conditions can also change between proposal and execution.

Model alignment matters, but so does the decision layer that determines whether a proposed action should proceed, be verified, rerouted, or escalated. Sayash Kapoor and Arvind Narayanan make the point directly in their recent essay on loss-of-control incidents:

“Model alignment is not enough for AI safety. To recap: when a model is deployed, it doesn’t have enough context to know if the conditions it’s being deployed in are safe or not — a lot of that context is outside the model.”

Instead of alignment as everything, they describe four complementary layers:

• Alignment: shaping model behavior.
• Control: constraining and monitoring deployed AI systems.
• Downstream defense: protecting the systems AI can affect.
• Resilience: limiting harm and enabling recovery after failures.

Their conclusion is that “marginal investments in control are more likely to be effective compared to those in alignment,” and that control has to keep pace with capability. We agree, and this is exactly where our focus is. Themis’s decision-layer platform sits in the control layer, between a model’s output and the action that follows. It connects that output, the available evidence and the operating context to the right next step. Capsa, our risk-estimation technology, supplies one of the key signals: how far each output can be trusted. It derives risk signals from internal model behavior, helping identify unfamiliar inputs and unreliable outputs. The context that is missing from the model is what a decision layer brings back into the loop.

Figure 1: The four layers of AI safety described by Kapoor and Narayanan, from the model outward. Themis works in the control layer, with task-specific connections to downstream defense. The layer names are theirs; the placement is ours.

The decision layer

Models are increasingly tasked with proposing actions or “tool calls”, but that doesn’t mean they should always be executed. Before consequential action, a decision layer establishes whose authority applies, what the user intended and consented to, which policies govern, and whether trusted evidence supports acting now. Then it chooses the next step: whether to proceed autonomously, verify the action, reroute to another model or tool, or escalate to a human expert. Only the proceed route releases the proposed action for execution. The other routes withhold execution while seeking evidence or an alternative; an action that remains unsupported or prohibited does not proceed.

Figure 2: A decision layer between a proposal and its effects: authority, intent and consent, governing policy and trusted evidence, then one of four routes. Outcomes feed the next decision.

Hard permissions, sandboxing and deterministic rules are essential foundations; the decision layer complements them. Some of those controls already use context, such as AWS’s temporal authorization policies, which check prior tool activity, data freshness and approval events. What a permission check alone cannot tell you is how much to trust a particular model output. That is the signal Capsa adds, helping determine whether a permitted action should proceed now, await more evidence, or be handed off to another model or a human expert. A risk estimate never creates authority or overrides a prohibition; it informs the choice between the routes that policy allows.

Whether an action should proceed depends on current authority, evidence and operating conditions. An agent arranging a service within a spending limit should stop when that limit cannot be established, should not treat a generic “go ahead” as consent to resolve a conflict the user was never shown, and should reassess if the price or other material terms change after approval. These checks must be connected to actual execution; an advisory warning that the agent can route around does not provide that control. Nor do these rules belong in the weights, where every policy change means another training run.

Review only the risky cases

Kapoor and Narayanan expect control to matter more as agents improve, not less. Agents’ reliability will keep lagging their capability, so frontier tasks will keep needing human supervision. And where errors carry high costs, they argue, even rare failures have to be prevented, which makes human supervision worth more than whatever full autonomy would gain.

That second point is the dilemma our customers often find themselves in. When an error is expensive, the default control is a person in front of every output. This defeats much of the purpose of what the automation was built for, and the system fails anyway: as the essay notes, approving every action “quickly leads to ‘alarm fatigue’ and the approval becomes a rubber-stamping exercise.” NIST warns of the same consent fatigue in agentic systems.

Risk-based review is the way out of this dilemma, and it is what Themis provides. Capsa estimates how much to trust each output so that the decision layer can pass confident cases through and route the uncertain, unfamiliar or high-stakes ones to verification, to another model or tool, or to an expert. Reviewers see fewer cases, and the cases they see are the ones that need them. That is the pattern behind our customer work: automate the easy majority and focus expert review where it changes the outcome. The decision layer carries that pattern through to action, enforcing the policy rather than just displaying a score.

The same principle applies beyond tool use: whether extracted information is reliable enough to update a record, whether a prediction needs another measurement, or whether changing inputs warrant a fresh check before proceeding.

How we measure it

Our test for a decision layer is simple. We take a strong baseline with all of its existing controls and compare it with the same system plus the decision layer, on the same tasks under the same conditions, counting unsafe or unauthorized actions, successful authorized completions, expert-review burden, latency and cost. Unauthorized actions are not a metric to be traded against completion rate; a prohibited action counts as a failure regardless of its outcome. Then we change the conditions on purpose, because inputs, policies and system state that differ from development are where a decision layer really demonstrates its value. In production, every decision should leave an inspectable record of the evidence, authority, policy, chosen response and outcome, which is what lets a team investigate failures, reassess as conditions change, and answer auditors. If ordinary controls do just as well for a task, we simply keep them.

Start with one consequential decision

Our ambition is to be the decision layer between AI and the actions taken on its outputs, across models, tools and workflows. We build it the way control gets built anywhere: one bounded, task-specific workflow at a time, where authority, evidence, conditions and outcomes can be measured.

If you run AI where errors are expensive and review is heavy, pick one consequential decision, its authority boundary, and what failures and review currently cost you. That gives us a concrete question to answer together: can better decisions before execution keep the useful work and remove the unacceptable outcomes?

Tell us about that decision: get in touch.