About a-team Marketing Services
The knowledge platform for the financial technology industry
The knowledge platform for the financial technology industry

A-Team Insight Blogs

How to Get Data Lineage Right – Key Challenges

Subscribe to our newsletter

Rolling out data lineage remains a challenge for financial institutions. How should data management teams approach lineage and put a framework in place to ensure it is complete and crosses business lines to meet the goals of data value?

A panel discussion during A-Team Group’s recent Data Management Summit held in the City of London on March 21, 2019, covered the key challenges in addressing data lineage and delved deep into the reasons why businesses need to urgently address data lineage strategies to improve efficiency.

The panel was moderated by Nicola Askham, The Data Governance Coach, and joined by Naomi Clarke, former head of data at GAM; Barry Green, former chief data officer at Bank of Ireland; Stephen Veasey, CEO at 3D Innovations; Ian Evans, managing director of EMEA at OneTrust; and Nimrod Vax, co-founder and head of product at Big ID.

Discussion points included the impact of business drivers and regulations such as General Data Protection Regulation (GDPR) on data lineage; the question of how firms can scale up data lineage programmes; the journey from a manual to automated approach; the impact of new and emerging technologies; and best practices for sustaining data lineage once documented.

“It’s not just about the data, but about the vendors,” stressed one panellist. “How do we recruit them, how do we onboard them? Data lineage brings that journey to life.”

While the panel agreed that a coherent data lineage programme is critical, the concern was that a large number of organisations simply have not yet thought through or addressed the issue. “We still struggle with the basics,” said one panellist. “We urgently need to know where our data is coming from. That’s crucial if we are going to start using it to inform decisions in data analytics.”

Concerns include GDPR and data licensing, which has become increasingly complex, making data lineage processes even more important. “You need to know where your data is from in order to be able to comply. Surveillance and the ability to declare your data honestly is crucial to being able to comply with regulations,” said a panel member.

So what are the key challenges?

“It’s not something you just install out of a box – it has to be a coherent solution,” said one panellist. “You have to know your business environment and understand what you are trying to achieve within the data universe. Most financial institutions do not have comprehensive enterprise-wide digital rights management capabilities the way they might have in other industries, like music or entertainment. Many companies are struggling with inherited legacy systems, multiple tech stacks and brownfield systems. That’s a real challenge, and it pushes you into addressing the issue on a case-by-case basis, which makes scalability a concern.”

It is a complex environment – and one solution is improved communication between business and IT functions. “There needs to be a combined approach where one enriches the other – that is the key challenge,” said a panellist.

As one panellist pointed out, data lineage shouldn’t be a regulatory solution: “It should be a solution for the business. Implementing good data management solutions allows the business to become more agile.” “There is no point doing data lineage if it’s static. You need some form of ownership, so that the person responsible makes sure it is fit for purpose,” noted another. “Data is dynamic, so you need to move towards a dynamic process where business users are informed along the way as needs, requirements and uses change. That will make the processes and data flows more relevant and continuously compliant, you can automatically identify changes and trigger interaction with business users to get a response.”

Keep it simple

“Data lineage has been around for a long time – it is nothing new,” noted a panel member. “Keep it simple. You don’t have to spend months and months documenting a 1,500 step process. The organisation needs to be able to look at the data, understand it, consume it and process it. Only add detail as and when you need it.”

“Don’t over complicate the data you capture,” agreed another. “In the event of a breach, lineage gives us the information about where the breach was, what caused it, how did it happen and how you can stop it happening again. Lineage is a visible diagram, it is a proactive tool that allows us to unlock questions about our business. That is its purpose.”

So where are we now in terms of automating data lineage? A survey undertaken during the panel found that while 37% of respondents at the Summit were planning data lineage automation, just 5% had made any significant progress and 21% had not made any progress at all. The biggest challenges were identified as data fragmentation (47%), no budget/lack of resources (22%), and a lack of business understanding (15%).

Subscribe to our newsletter

Related content

WEBINAR

Upcoming Webinar: Mastering Data Lineage for Risk, Compliance, and AI Governance

18 June 2025 10:00am ET | 3:00pm London | 4:00pm CET Duration: 50 Minutes Financial institutions are under increasing pressure to ensure data transparency, regulatory compliance, and AI governance. Yet many struggle with fragmented data landscapes, poor lineage tracking and compliance gaps. This webinar will explore how enterprise-grade data lineage can help capital markets participants...

BLOG

S&P’s Capital IQ Pro Flagship Offers Visible Alpha with View to Expansion

Behind every stock analyst’s financial forecast report is a detailed data model that tracks the company so that they can arrive at a view on that asset. Until about eight years ago it was difficult to prise open those reports to understand the thinking behind the headline numbers. Visible Alpha helped change that. Founded by...

EVENT

TradingTech Briefing New York

Our TradingTech Briefing in New York is aimed at senior-level decision makers in trading technology, electronic execution, trading architecture and offers a day packed with insight from practitioners and from innovative suppliers happy to share their experiences in dealing with the enterprise challenges facing our marketplace.

GUIDE

AI in Capital Markets: Practical Insight for a Transforming Industry – Free Handbook

AI is no longer on the horizon – it’s embedded in the infrastructure of modern capital markets. But separating real impact from inflated promises requires a grounded, practical understanding. The AI in Capital Markets Handbook 2025 provides exactly that. Designed for data-driven professionals across the trade life-cycle, compliance, infrastructure, and strategy, this handbook goes beyond...