The ingestion of growing volumes of data into financial institutions’ systems is posing a pressing challenge as data managers seek to optimise their data lineage, according to the latest A-Team Group webinar.
Being able track data as it enters and is distributed within organisations is essential for prising the most value from that information. However, as organisations take on more data feeds to satisfy their processes’ information demands, it has become difficult to identify accountable parties or even where to commence lineage initiatives.
This underscores the need to capture not just data stores, but all surrounding processes, from SQL scripts to BI transformations, the webinar, entitled “Mastering Data Lineage for Risk, Compliance, and AI Governance”, heard.
Challenges to lineage optimisation were among the key themes discussed by the webinar panel, which comprised Viktor Godaly, Head of Group Data Governance and Data Control at Danske Bank; Mark McQueen, Managing Partner at Ortecha; and, Christian Bremeau, Chief Executive of Meta Integration, the event’s sponsor.
Many Challenges
Moderated by A-Team Group Editor Sarah Underwood, the chief challenges were highlighted by a poll of viewers, drawn from capital markets participants and data leaders. Four-fifths of respondents cited multiple data sources as the biggest challenge to their data lineage management. That was followed closely by data siloes, which are a problem for two-thirds of respondents, while legacy systems – some of which date back to the 1970s, the webinar heard – pose integration issues.
The depth and completeness of lineage trouble many firms, too, the panellists said, adding that many existing products built to counter the problem are deemed “shallow” and fail to provide feature-level or column-level detail. They also omit critical transformations within various tools.
The complexity of change management was also highlighted, with the panel agreeing that aligning the business side with the need to improve lineage is an ongoing struggle.
No matter the challenges, they have to be overcome to ensure full transparency and integration across diverse technologies so that compliance and other processes can be executed efficiently and effectively.
Implementation Gap
This is no easy ask, the webinar heard. Another poll found that a small minority – just 7 per cent of respondents – had implemented enterprise-grade data lineage to a great extent in order to improve risk, compliance and AI governance. The single-largest response came from two-fifths of respondents who said they had implemented enterprise-grade data lineage to a good extent.
This bore out the comment from one panellist, who said that end-to-end lineage across an entire infrastructure remains “rather limited” across financial institutions.
Panellists also agreed that data lineage had utility beyond keeping data in order so that it can meet compliance obligations. Treating lineage as a fundamental risk control, integrated into risk registers and compliance dashboards, was strongly advocated. This aligns with a shift towards design observability – understanding what should happen to data based on policies and standards, recorded as machine-readable metadata – to be combined with data observability, which tracks what actually happens. This pairing allows for demonstrable compliance.
AI Imperative
For AI governance, lineage is foundational, the panel said. It ensures trusted data inputs for model training and operations, clarifying data origin, quality and ownership. While AI models can produce answers, the underlying logic often remains opaque, unlike traditional BI reports where every formula can be traced. Lineage, therefore, becomes essential for verifying the quality and provenance of the data feeding the AI.
Subscribe to our newsletter
