The leading knowledge platform for the financial technology industry
The leading knowledge platform for the financial technology industry

A-Team Insight Blogs

How to Get Data Lineage Right – Key Challenges

Subscribe to our newsletter

Rolling out data lineage remains a challenge for financial institutions. How should data management teams approach lineage and put a framework in place to ensure it is complete and crosses business lines to meet the goals of data value?

A panel discussion during A-Team Group’s recent Data Management Summit held in the City of London on March 21, 2019, covered the key challenges in addressing data lineage and delved deep into the reasons why businesses need to urgently address data lineage strategies to improve efficiency.

The panel was moderated by Nicola Askham, The Data Governance Coach, and joined by Naomi Clarke, former head of data at GAM; Barry Green, former chief data officer at Bank of Ireland; Stephen Veasey, CEO at 3D Innovations; Ian Evans, managing director of EMEA at OneTrust; and Nimrod Vax, co-founder and head of product at Big ID.

Discussion points included the impact of business drivers and regulations such as General Data Protection Regulation (GDPR) on data lineage; the question of how firms can scale up data lineage programmes; the journey from a manual to automated approach; the impact of new and emerging technologies; and best practices for sustaining data lineage once documented.

“It’s not just about the data, but about the vendors,” stressed one panellist. “How do we recruit them, how do we onboard them? Data lineage brings that journey to life.”

While the panel agreed that a coherent data lineage programme is critical, the concern was that a large number of organisations simply have not yet thought through or addressed the issue. “We still struggle with the basics,” said one panellist. “We urgently need to know where our data is coming from. That’s crucial if we are going to start using it to inform decisions in data analytics.”

Concerns include GDPR and data licensing, which has become increasingly complex, making data lineage processes even more important. “You need to know where your data is from in order to be able to comply. Surveillance and the ability to declare your data honestly is crucial to being able to comply with regulations,” said a panel member.

So what are the key challenges?

“It’s not something you just install out of a box – it has to be a coherent solution,” said one panellist. “You have to know your business environment and understand what you are trying to achieve within the data universe. Most financial institutions do not have comprehensive enterprise-wide digital rights management capabilities the way they might have in other industries, like music or entertainment. Many companies are struggling with inherited legacy systems, multiple tech stacks and brownfield systems. That’s a real challenge, and it pushes you into addressing the issue on a case-by-case basis, which makes scalability a concern.”

It is a complex environment – and one solution is improved communication between business and IT functions. “There needs to be a combined approach where one enriches the other – that is the key challenge,” said a panellist.

As one panellist pointed out, data lineage shouldn’t be a regulatory solution: “It should be a solution for the business. Implementing good data management solutions allows the business to become more agile.” “There is no point doing data lineage if it’s static. You need some form of ownership, so that the person responsible makes sure it is fit for purpose,” noted another. “Data is dynamic, so you need to move towards a dynamic process where business users are informed along the way as needs, requirements and uses change. That will make the processes and data flows more relevant and continuously compliant, you can automatically identify changes and trigger interaction with business users to get a response.”

Keep it simple

“Data lineage has been around for a long time – it is nothing new,” noted a panel member. “Keep it simple. You don’t have to spend months and months documenting a 1,500 step process. The organisation needs to be able to look at the data, understand it, consume it and process it. Only add detail as and when you need it.”

“Don’t over complicate the data you capture,” agreed another. “In the event of a breach, lineage gives us the information about where the breach was, what caused it, how did it happen and how you can stop it happening again. Lineage is a visible diagram, it is a proactive tool that allows us to unlock questions about our business. That is its purpose.”

So where are we now in terms of automating data lineage? A survey undertaken during the panel found that while 37% of respondents at the Summit were planning data lineage automation, just 5% had made any significant progress and 21% had not made any progress at all. The biggest challenges were identified as data fragmentation (47%), no budget/lack of resources (22%), and a lack of business understanding (15%).

Subscribe to our newsletter

Related content

WEBINAR

Upcoming Webinar: Key steps in adopting cloud and SaaS delivery for enterprise data

Date: 29 March 2022 Time: 10:00am ET / 3:00pm London / 4:00pm CET Duration: 50 minutes Following in the footsteps of market data migration to the cloud, enterprise data is finding its place in the cloud alongside Software-as-a-Service apps and data delivery mechanisms. The initial aim is to achieve greater efficiency and reduced costs by...

BLOG

A-Team Group Names Winners of Data Management Insight Awards 2021

A-Team Group has named the winners of its prestigious 2021 Data Management Insight Awards. The annual awards, now in their ninth year, recognise leading providers of data management solutions, services and consultancy to capital markets participants. Award categories ranged from best data lineage solution to best data quality analysis tool, best standards solution, best corporate...

EVENT

RegTech Summit London

Now in its 7th year, the RegTech Summit in London will bring together the RegTech ecosystem to explore how the European capital markets financial industry can leverage technology to drive innovation, cut costs and support regulatory change.

GUIDE

Regulatory Data Handbook 2021/2022 – Ninth Edition

Welcome to the ninth edition of A-Team Group’s Regulatory Data Handbook, a publication dedicated to helping you gain a full understanding of regulations related to your organisation from the details of requirements to best practice implementation. This edition of the handbook includes a focus on regulations being rolled out to bring order and standardisation to...