The leading knowledge platform for the financial technology industry
The leading knowledge platform for the financial technology industry

A-Team Insight Blogs

How to Get Data Lineage Right – Key Challenges

Rolling out data lineage remains a challenge for financial institutions. How should data management teams approach lineage and put a framework in place to ensure it is complete and crosses business lines to meet the goals of data value?

A panel discussion during A-Team Group’s recent Data Management Summit held in the City of London on March 21, 2019, covered the key challenges in addressing data lineage and delved deep into the reasons why businesses need to urgently address data lineage strategies to improve efficiency.

The panel was moderated by Nicola Askham, The Data Governance Coach, and joined by Naomi Clarke, former head of data at GAM; Barry Green, former chief data officer at Bank of Ireland; Stephen Veasey, CEO at 3D Innovations; Ian Evans, managing director of EMEA at OneTrust; and Nimrod Vax, co-founder and head of product at Big ID.

Discussion points included the impact of business drivers and regulations such as General Data Protection Regulation (GDPR) on data lineage; the question of how firms can scale up data lineage programmes; the journey from a manual to automated approach; the impact of new and emerging technologies; and best practices for sustaining data lineage once documented.

“It’s not just about the data, but about the vendors,” stressed one panellist. “How do we recruit them, how do we onboard them? Data lineage brings that journey to life.”

While the panel agreed that a coherent data lineage programme is critical, the concern was that a large number of organisations simply have not yet thought through or addressed the issue. “We still struggle with the basics,” said one panellist. “We urgently need to know where our data is coming from. That’s crucial if we are going to start using it to inform decisions in data analytics.”

Concerns include GDPR and data licensing, which has become increasingly complex, making data lineage processes even more important. “You need to know where your data is from in order to be able to comply. Surveillance and the ability to declare your data honestly is crucial to being able to comply with regulations,” said a panel member.

So what are the key challenges?

“It’s not something you just install out of a box – it has to be a coherent solution,” said one panellist. “You have to know your business environment and understand what you are trying to achieve within the data universe. Most financial institutions do not have comprehensive enterprise-wide digital rights management capabilities the way they might have in other industries, like music or entertainment. Many companies are struggling with inherited legacy systems, multiple tech stacks and brownfield systems. That’s a real challenge, and it pushes you into addressing the issue on a case-by-case basis, which makes scalability a concern.”

It is a complex environment – and one solution is improved communication between business and IT functions. “There needs to be a combined approach where one enriches the other – that is the key challenge,” said a panellist.

As one panellist pointed out, data lineage shouldn’t be a regulatory solution: “It should be a solution for the business. Implementing good data management solutions allows the business to become more agile.” “There is no point doing data lineage if it’s static. You need some form of ownership, so that the person responsible makes sure it is fit for purpose,” noted another. “Data is dynamic, so you need to move towards a dynamic process where business users are informed along the way as needs, requirements and uses change. That will make the processes and data flows more relevant and continuously compliant, you can automatically identify changes and trigger interaction with business users to get a response.”

Keep it simple

“Data lineage has been around for a long time – it is nothing new,” noted a panel member. “Keep it simple. You don’t have to spend months and months documenting a 1,500 step process. The organisation needs to be able to look at the data, understand it, consume it and process it. Only add detail as and when you need it.”

“Don’t over complicate the data you capture,” agreed another. “In the event of a breach, lineage gives us the information about where the breach was, what caused it, how did it happen and how you can stop it happening again. Lineage is a visible diagram, it is a proactive tool that allows us to unlock questions about our business. That is its purpose.”

So where are we now in terms of automating data lineage? A survey undertaken during the panel found that while 37% of respondents at the Summit were planning data lineage automation, just 5% had made any significant progress and 21% had not made any progress at all. The biggest challenges were identified as data fragmentation (47%), no budget/lack of resources (22%), and a lack of business understanding (15%).

Related content

WEBINAR

Recorded Webinar: Getting ready for Sustainable Finance Disclosure Regulation (SFDR) and ESG – what action should asset managers be taking now?

Interest in Environmental, Social and Governance (ESG) investment has exploded in recent years, bringing with it regulation and a requirement for buy-side firms to develop ESG strategies and meet disclosure obligations. The sell-side can help here by integrating ESG data with traditional financial information, although the compliance burden remains with asset managers. The EU Sustainable...

BLOG

DSB Opens First Consultation on UPI Fee Principles

The Derivatives Service Bureau (DSB) opened a first round of industry consultation on fee principles for the Unique Product Identifier (UPI) today. The consultation ends on 5 March 2021 and is the first of two consultations before a final report is published in September 2021. The UPI will come into play in 2022 and will...

EVENT

Data Management Summit Virtual

The Data Management Summit Virtual will bring together the global data management community to share lessons learned, best practice guidance and latest innovations to emerge from the recent crisis. Join us online to hear from leading data practitioners and innovators from the UK, US and Europe who will share insights into how they are pushing the boundaries with data to deliver value with flexible but resilient data driven strategies.

GUIDE

Entity Data Management Handbook – Seventh Edition

Sourcing entity data and ensuring efficient and effective entity data management is a challenge for many financial institutions as volumes of data rise, more regulations require entity data in reporting, and the fight again financial crime is escalated by bad actors using increasingly sophisticated techniques to attack processes and systems. That said, based on best...