Debate about the limitations of artificial intelligence (AI) in data management was stoked further this week when a leading vendor warned that applications built on nascent large language model (LLM) technology could pose an “existential threat” to companies if not deployed thoughtfully.
Jason du Preez, vice president of privacy and security at cloud data management provider Informatica, said that the growing use of LLMs needed to be accompanied by strong governance and privacy controls to ensure organisations were not harmed by the technology’s potential unpredictability.
“Inadvertently leaking or exposing sensitive information, can pose an existential threat to organisations leading to massive reputational damage and economic loss – this includes data not consented by individuals, a risk exacerbated by the use of nascent LLM-based applications that can produce unpredictable outputs,” said du Preez, whose company provides AI-based data tools to financial institutions and other industries around the world.
Without sound data governance practices and appropriate security and privacy controls, few generative AI (GenAI) projects will get off the ground, du Preez said in comments timed to coincide with Data Privacy Week.
“The fact that data flows are increasingly complex across hyper-scalers and on-premises data centres makes it impossible to manage without systems and processes that span all these ecosystems,” du Preez said. “Delivering end-to-end governance across eco-systems along with clear data security and privacy controls is critical to providing a solid foundation for GenAI.”
Taking Stock
The initial rush of investment into AI following the headline-grabbing release of open-source GenAI application ChatGPT two years ago is expected to ease as companies balance the technology’s costs against its benefits. In Data Management Insight’s latest survey of expert predictions for the coming year, several observers said they expect some of the froth to come off the AI market in 2025.
Informatica chief industry strategist Peter Ku told Data Management Insight that many banks were struggling to work out how best to scale AI and make it fit onto their budgets. For some, AI transformation may not be happening at the pace they’d hoped, Ku said.
Similarly, Akber Jaffer, chief executive of data automation provider SmartStream, said that some of his company’s clients were taking a more cautious approach to LLM adoption.
The cost of the technology and the difficulty of implementing and managing an AI strategy from scratch are among the impediments that early adopters have cited, while some have compared the surge of investment into AI with the exuberance that resulted in the bursting of the dot-com bubble in 2000.
AI Uncertainty
The latest warning comes amid a backdrop of market volatility driven by a reappraisal of the strength of US-based models and the makers of the hardware that runs AI. The emergence of Chinese AI start-up DeepSeek’s low-cost model as a competitor to the likes of Meta and Anthropic has prompted a rethink of how AI will be used, potentially lowering the barrier to entry for smaller companies.
In Europe, however, the implementation of the EU’s AI Act could put a drag on AI adoption with its requirements that companies ensure they have adequate AI literacy among their employees. It will also prohibit what it calls unacceptable risk AI and expect organisations to provide technical documentation and instructions on the use of models, including LLMs.
“It’s promising that LLMs are put under real scrutiny, as they are more likely to play host to misinformation, infringe on the public’s privacy and create complex new questions surrounding copyright law,” said Laura Petrone, a principal analyst at market data provider GlobalData.
Protective Measures
Du Preez argues that while AI could pose difficulties to data teams, such issues needn’t arise if implementation is preceded by careful planning and accompanied by a safety net.
“Robust data privacy and governance frameworks aren’t just compliance checkboxes – they’re the bedrock of responsible AI development and deployment,” du Preez said. “Organisations that prioritise privacy-centric data governance are better positioned to navigate the evolving landscape of AI regulations while maintaining the trust of their stakeholders.”
Subscribe to our newsletter
