As investment firms move AI agents out of experimental environments and into live workflows, the governance problem is becoming harder to ignore.
Much of the early discussion around agentic AI has focused on what agents can do: retrieve information, call tools, analyse data, generate recommendations and increasingly act on behalf of users. Once those agents start operating inside investment and trading workflows, firms also need to know exactly what happened when something goes wrong.
Can they reconstruct the sequence of events? Can they work out why a particular decision was made? Can they show that the outcome matched the intended policy or mandate? And can they defend that decision to regulators, auditors or clients?According to Yiannis Antoniou, CEO of Lydatum, the AI & Data expert advisory and implementation services company, observability is becoming a basic requirement for putting agentic systems into live workflows.
“If you move an agent into production,” he explains to TradingTech Insight, “you need a way to observe and instrument it and be able to say: it’s doing this, it’s doing that, it’s not behaving the way I expected, or it’s deviating from what I saw during the pilot. This is becoming non-negotiable for production implementation.”
From observability to explainability
At the most basic level, firms are getting better at recording what agents actually do. A growing range of AI observability tools can capture tool calls, prompts, parameters, outputs and intermediate results, creating traces that let firms reconstruct how an agent moved through a task. Some platforms can preserve those traces at a granular level so they can later be replayed or audited.
This is also starting to converge with established observability and telemetry technologies. Traditional monitoring vendors are adding AI-specific capabilities, while newer platforms are being built around agent behaviour. AI gateways add another layer by logging interactions, routing requests across models and monitoring costs alongside operational behaviour.A detailed trace can show that an agent queried a particular dataset, called a particular model and then triggered another tool. However, it may still leave the firm with a harder problem: understanding why the agent reached the conclusion it did and whether that conclusion was appropriate.
“We may be able to trace some of the inner mechanisms of what agents do, but we don’t really have a universal way of saying why they did it,” notes Antoniou. “We need to prove not just what agents are doing, but whether what they are doing is actually what we intended them to do.”
That gap is likely to become more visible as agents take on more consequential roles in investment workflows.
The problem of non-determinism
Traditional software auditing generally assumes that if the same inputs pass through the same logic, the system should behave in the same, consistent way. However, generative AI doesn’t always work like that. Because large language models and agentic systems are inherently non-deterministic, replaying the same sequence doesn’t necessarily reproduce exactly the same outcome.
“We have auditability for a particular invocation of a particular agent, but we don’t have guarantees that it will do exactly the same thing again under the same circumstances when we invoke it again,” says Antoniou. “That hasn’t been solved yet, and I don’t know that we can solve it 100%, given the nature of the tools themselves.”
For investment firms, this creates practical problems around model monitoring and incident investigation. If an agent makes a poor recommendation, executes an inappropriate action or behaves differently from the way it did during testing, the firm needs enough evidence to diagnose what happened and understand whether the behaviour can be reproduced or corrected.
The problem gets harder in multi-agent environments, where one agent may pass information or instructions to another, creating long chains of activity that need to be correlated across systems.
Evals help, but they don’t close the gap
One response is the growing use of evaluations, or “evals”, to test whether agent outputs line up with expected behaviour. In many cases, those evaluations themselves use LLMs as judges, comparing an output with the original prompt or intended task and producing a confidence score. That can provide useful assurance, but it remains probabilistic. A firm may, in effect, be asking one non-deterministic system to assess another.
As Antoniou points out: “If I am a highly regulated bank and I want to prove that every action taken on behalf of a user is not just fully traceable but fully explainable, I don’t know that I have a 100% ironclad guarantee today that would survive in court.”
There are other governance complications too. Detailed traces may contain personally identifiable information or other sensitive data, so the audit trail itself needs masking, access controls and retention policies. Correlating traces across multiple agents and systems adds another layer of difficulty.
Building governance into the architecture
The technology is improving quickly, but there is still no single standard or architecture that gives firms end-to-end assurance.
Antoniou expects more standardisation to emerge, potentially through extensions to existing observability frameworks such as OpenTelemetry. Even so, better logging on its own won’t answer every governance question.
For firms putting agents into production, the practical requirement is likely to be layered: trace what the agent did, evaluate whether its actions matched the intended objective, preserve enough evidence for review, and keep human accountability clear.
As agents move further into investment and trading workflows, firms will need to be able to show not only how an action happened, but why they were prepared to let the agent take it in the first place.
Subscribe to our newsletter
