Rush to Build AI into Risk Models Creating Dangerous Capability ‘Gaps’

Asset managers may be exposing themselves to operational vulnerabilities as they rapidly integrate artificial intelligence and agents into their risk management systems, often without first ensuring trust in the data that will feed the models.

A survey found that three-quarters of 178 senior investment leaders questioned said they expect the pace of AI and agents deployment in their risk operations to gather pace in the next three years. They almost unanimously proclaimed their confidence in the technology, too.

“As AI moves deeper into risk management, the quality of the underlying data is becoming the strongest differentiator among firms, more than the sophistication of the AI itself,” Viverito told Data Management Insight.

“Data trust will determine which firms turn AI investment into genuinely better decisions.”

Value Tested

As the deployment of AI and agents has become the over-arching story in FinTech over the past two years, studies like those of Clearwater Analytics’ are highlighting the challenges and potential dangers firms are facing as they race to incorporate a technology whose value is still being tested.

Reports of rogue AI models and hallucinations have fostered an atmosphere of distrust within the general population that isn’t fully reflected in the financial industry. But influential voices have nonetheless urged a more considered approach to adoption.

The Durham University Business School, for instance, has called for specific regulations over AI to limit risks to consumers and strengthen oversight across the financial sector.

Growing Threat

The risks posed by the data-trust gap will be amplified as more capital is invested in AI and agents. According to the Clearwater report, four in five managers expect to increase capital expenditure on the technology by at least half over the next year. Almost two thirds said they’re committed to a more-than 50% increase, while a fifth said their budgets would swell by as much as 300%. Just 5% said they would hold or reduce their financial outlay.

The central concern – whether data used by AI models and agents is trustworthy – dominated discussion at A-Team Group’s AI in Capital Markets Summit in London this year. Invited speakers and experts noted that the criticality of data quality had increased as agents automated tasks with little to no human oversight.

The use of old and stale data, for instance, risks creating a domino effect of processing errors that could ripple downstream through the entire enterprise, delegates heard. The implications of such viral spreads could result in poor decision making that would lead to lost business and lost customers, and it would raise the risk of compliance breaches that would attract potentially heavy fines and reputational damage.

Human Touch

Concern around the retreat of humans from the oversight loop amid agentic adoption was highlighted in a study commissioned by hedge fund manager Nickel Digital Asset Management. It found that institutional investors and wealth managers would like to see greater use of AI in digital asset investment processes so long as humans retained decision oversight.

The report said almost nine in 10 of the 203 institutional investment leaders interviewed across the world said they would be comfortable using AI in their investment processes subject to humans remaining in the loop.

Anatoly Crachilov, CEO and Founding Partner at Nickel Digital, said that “there is a growing consensus that an AI-led investment process is the direction of travel, but for managers, human oversight remains non-negotiable”.

The Nickel findings reflect a broader anxiety over the lack of human oversight of agents. Data and AI trust company Veeam Software found in its own recently published research that organisations across all industries in the Europe, Middle East and Africa region expressed concern about unmanaged AI creating a “shadow agents” crisis.

More than two-thirds of those questioned aired fears that automated workflows are interacting with sensitive corporate data without full oversight. Also, they expressed alarm over a perception that IT departments were losing control because employees were creating agentic workflows that couldn’t be tracked by their internal systems.

In its report, which concluded that governance had become a C-suite liability, Veeam said shadow agents were spreading across the region faster than firms’ oversight policies could match.

“Executives have never had more to think about around data governance and cyber resilience regulations,” said Tim Pfaelzer, General Manager and Senior Vice President for EMEA at Veeam. “While the increased stress and tension are completely understandable, if the result is better alignment at the board level, the organisation will be far better positioned to trust its data and AI governance in the agentic era.

“With greater alignment ultimately comes better data governance, visibility and protection across the organisation,” Pfaelzer added.

Caution Control

The findings of the Clearwater Analytics research have added to the growing sense of anxiety because risk management is considered the most sensitive part of an institution’s operations, acting as the gatekeeper keeping a watch on poor decision-making.

“One would expect the most cautious part of the business to move the slowest on something new; this data says the opposite. Firms are leaning into AI in one place they can least afford to get wrong,” said Souvik Das, Chief Technology Officer at Clearwater.

“Risk management is where a firm’s data has nowhere to hide. A slow report is forgivable. A risk signal built on bad data isn’t… that [data quality] gap is what decides whether a risk signal can be trusted. The firms closing it are the ones making sharper decisions, with better information than they’ve had before.”