For Philip Dutton, chief executive of data lineage specialist Solidatus, operating a business whose data dependencies have no robust lineage controls is like boarding a flight on an aeroplane that has no blueprints.
“If they told you they didn’t know how the plane was put together, I’m not sure you’d step on board,” Dutton tells Data Management Insight from his office in Texas.
“We operate businesses without that blueprint every day but we should understand deeply how our organisation operates and the things that keep it flying versus the things that are likely to cause it to crash.”
To extend the playful metaphor, without good data lineage management, companies are flying blind into potential compliance breaches, poor AI outcomes, inaccurate reporting and bad leadership decisions.
Now or Never
Understanding that has never been more important, argues Dutton, as financial institutions have embarked upon a profound transitional journey built on AI and agentic AI, transforming the way many operate and do business. It’s a critical consideration, too, as they undergo a period of unprecedented data consumption and generation that has fundamentally reshaped the way they manage that data and how they engage with regulators.
This period of change has given lineage its day in the sun, he says.
“What AI has really done is demonstrated the scale at which we can do things much more quickly and the speed at which that can occur,” he explains.
Before now, companies paid mere lip service to their metadata but the demands of managing huge volumes of enterprise data and the need to get that data right for AI applications and regulators has brought lineage to the forefront of chief data officers’ minds, says Dutton.
When things go wrong now, they can affect entire enterprises and not just a single user or department.
“That’s the real determining factor of why people are sitting up and taking notice.”
Halt in the Road
Disruptions caused by weak lineage have become more prevalent in recent years as data complexity and regulatory requirements have intensified.
That’s been particularly visible in compliance with the BCBS 239 regulation that obliges institutions to take steps to secure and ensure optimal quality of their risk data capabilities and risk reporting. The Bank of International Settlements said earlier this year that effective data traceability remains a challenge for institutions seeking to meet the regulation’s requirements.
“Legacy systems, distributed data estates and the dynamic nature of data lineage complicate banks’ efforts to confirm end-to-end data traceability,” the organisation wrote in a publication in January. “Finding appropriate vendor solutions and the resource-intensive nature of identifying and maintaining data lineage can further hinder progress.”
Dutton says he has seen companies struggle with other regulatory obligations, including compliance with DORA, as a result of having weak metadata management. He said banks of all sizes have also lost multiple trading days thanks to system updates that have failed because poor lineage has meant the update’s impact couldn’t be risk assessed across the enterprise prior to release.
“Basically the banks just stop,” he says. “Those people hadn’t realised that the change has had an impact six hops away at another system that wasn’t aware that the change was going on.”
Failed Pilots
The fundamental importance of lineage was underlined in a Gartner report that forecast four-fifths of all data and analytics governance initiatives would have failed last year because of a lack of effective metadata management.
While the industry is still in the upward phase of AI adoption, the technology’s potential and the treatment of data to feed it has made data leaders realise the importance of ensuring their metadata is in order.
“In the past we were able to live in a world where that information could exist in people’s heads; it became like institutional tribal knowledge,” he explains.
“But if they left, the black hole it created caused an organisational slowdown.”
Business Growth
Solidatus has had a busy few years, strengthening ties with platforms such as Databricks, Snowflake and Microsoft, and most recently launching an artificial intelligence agentic assistant to aid clients with their metadata management
The company’s core offering collects metadata to create a blueprint of organisations’ data dependencies. That’s been turbo-charged by the agentic assistants, which Dutton says significantly accelerate automation and modelling tasks, it can process unstructured data, and listen for signals from the data flow to notify users downstream of any potential impacts from metadata change or errors before they happen.
“It says ‘look, here are the standards and policies that I have set for my organisation and this is what good looks like from a metadata perspective’,” he says. “It can assess the metadata and make sure that nothing is missing.”
For the foreseeable future, Solidatus’ focus will remain on working with its platform partners, ensuring that a universal control plane can be integrated with them.
Further on the horizon, it hopes to make data lineage and metadata capture a seamless component of all operating models, with a concentration on ensuring decisions are made deterministically and not probabilistically in the pursuit of achieving full accuracy in lineage.
“We’d rather be 100 percent correct every time than 99 percent correct, 99 percent of the time,” he says. “That’s where you build trust. If we get things wrong once or twice, trust is degraded pretty quickly.”
Subscribe to our newsletter
