About a-team Marketing Services
The knowledge platform for the financial technology industry
The knowledge platform for the financial technology industry

A-Team Insight Blogs

How to Get Data Lineage Right – Key Challenges

Subscribe to our newsletter

Rolling out data lineage remains a challenge for financial institutions. How should data management teams approach lineage and put a framework in place to ensure it is complete and crosses business lines to meet the goals of data value?

A panel discussion during A-Team Group’s recent Data Management Summit held in the City of London on March 21, 2019, covered the key challenges in addressing data lineage and delved deep into the reasons why businesses need to urgently address data lineage strategies to improve efficiency.

The panel was moderated by Nicola Askham, The Data Governance Coach, and joined by Naomi Clarke, former head of data at GAM; Barry Green, former chief data officer at Bank of Ireland; Stephen Veasey, CEO at 3D Innovations; Ian Evans, managing director of EMEA at OneTrust; and Nimrod Vax, co-founder and head of product at Big ID.

Discussion points included the impact of business drivers and regulations such as General Data Protection Regulation (GDPR) on data lineage; the question of how firms can scale up data lineage programmes; the journey from a manual to automated approach; the impact of new and emerging technologies; and best practices for sustaining data lineage once documented.

“It’s not just about the data, but about the vendors,” stressed one panellist. “How do we recruit them, how do we onboard them? Data lineage brings that journey to life.”

While the panel agreed that a coherent data lineage programme is critical, the concern was that a large number of organisations simply have not yet thought through or addressed the issue. “We still struggle with the basics,” said one panellist. “We urgently need to know where our data is coming from. That’s crucial if we are going to start using it to inform decisions in data analytics.”

Concerns include GDPR and data licensing, which has become increasingly complex, making data lineage processes even more important. “You need to know where your data is from in order to be able to comply. Surveillance and the ability to declare your data honestly is crucial to being able to comply with regulations,” said a panel member.

So what are the key challenges?

“It’s not something you just install out of a box – it has to be a coherent solution,” said one panellist. “You have to know your business environment and understand what you are trying to achieve within the data universe. Most financial institutions do not have comprehensive enterprise-wide digital rights management capabilities the way they might have in other industries, like music or entertainment. Many companies are struggling with inherited legacy systems, multiple tech stacks and brownfield systems. That’s a real challenge, and it pushes you into addressing the issue on a case-by-case basis, which makes scalability a concern.”

It is a complex environment – and one solution is improved communication between business and IT functions. “There needs to be a combined approach where one enriches the other – that is the key challenge,” said a panellist.

As one panellist pointed out, data lineage shouldn’t be a regulatory solution: “It should be a solution for the business. Implementing good data management solutions allows the business to become more agile.” “There is no point doing data lineage if it’s static. You need some form of ownership, so that the person responsible makes sure it is fit for purpose,” noted another. “Data is dynamic, so you need to move towards a dynamic process where business users are informed along the way as needs, requirements and uses change. That will make the processes and data flows more relevant and continuously compliant, you can automatically identify changes and trigger interaction with business users to get a response.”

Keep it simple

“Data lineage has been around for a long time – it is nothing new,” noted a panel member. “Keep it simple. You don’t have to spend months and months documenting a 1,500 step process. The organisation needs to be able to look at the data, understand it, consume it and process it. Only add detail as and when you need it.”

“Don’t over complicate the data you capture,” agreed another. “In the event of a breach, lineage gives us the information about where the breach was, what caused it, how did it happen and how you can stop it happening again. Lineage is a visible diagram, it is a proactive tool that allows us to unlock questions about our business. That is its purpose.”

So where are we now in terms of automating data lineage? A survey undertaken during the panel found that while 37% of respondents at the Summit were planning data lineage automation, just 5% had made any significant progress and 21% had not made any progress at all. The biggest challenges were identified as data fragmentation (47%), no budget/lack of resources (22%), and a lack of business understanding (15%).

Subscribe to our newsletter

Related content

WEBINAR

Recorded Webinar: Unlocking value: Harnessing modern data platforms for data integration, advanced investment analytics, visualisation and reporting

Modern data platforms are bringing efficiencies, scalability and powerful new capabilities to institutions and their data pipelines. They are enabling the use of new automation and analytical technologies that are also helping firms to derive more value from their data and reduce costs. Use cases of specific importance to the finance sector, such as data...

BLOG

From Batch to Real-Time: LSEG Reinvents AML Screening with World-Check On Demand

As financial institutions accelerate toward real-time payments and digital onboarding, compliance teams face mounting pressure to keep customer screening instant, accurate and demonstrable. In response, the London Stock Exchange Group (LSEG) has introduced World-Check On Demand – a new cloud-based service designed to deliver “real-time risk intelligence” through API integration, allowing institutions to embed sanctions...

EVENT

AI in Capital Markets Summit London

The AI in Capital Markets Summit will explore current and emerging trends in AI, the potential of Generative AI and LLMs and how AI can be applied for efficiencies and business value across a number of use cases, in the front and back office of financial institutions. The agenda will explore the risks and challenges of adopting AI and the foundational technologies and data management capabilities that underpin successful deployment.

GUIDE

Regulatory Data Handbook 2025 – Thirteenth Edition

Welcome to the thirteenth edition of A-Team Group’s Regulatory Data Handbook, a unique and practical guide to capital markets regulation, regulatory change, and the data and data management requirements of compliance across Europe, the UK, US and Asia-Pacific. This year’s edition lands at a moment of accelerating regulatory divergence and intensifying data focused supervision. Inside,...