Data lineage is a hot topic, but just how hot and what should you be doing to ensure you get it right at your organisation? How can you ensure it delivers return on investment and is, indeed, an opportunistic solution?
A recent A-Team Group webinar answered these questions and more, underlining the importance of data lineage for both regulatory compliance and business strategy. It also answered numerous questions from listeners, highlighting industry interest in the topic, and pointed to A-Team’s recently released 2019 Data Lineage Handbook.
Giles Nelson, chief technology officer at MarkLogic, kicked off the conversation explaining how data lineage has developed from being a technical topic to a hot business topic over the past few years. He said: “We see many projects around regulatory reporting, but as data management and analytics have become more strategic to the business, lineage and the ability to trace data has become fundamental to all infrastructure.”
Considering the growing importance of data lineage, an audience poll on implementation progress showed 47% of participants having implemented lineage to the greatest extent possible, 33% significantly, 22% somewhat, a further 22% a little, and 18% not at all. Later polls during the webinar covered the challenges of deploying data lineage, and the operational and business benefits of successful programmes.
The webinar speakers reported pretty good progress at their organisations and in the market as a whole. Ellen Gentile, vice president and data quality manager at Sumitomo Mitsui Bank, said the bank was making great progress and had implemented a data glossary to support lineage. Dan Roberts, director of solutions engineering, UK and Ireland, at MarkLogic, noted firms extracting and storing metadata as a means of allowing users to query data lineage.
While progress is being made, the challenges of data lineage are many, Irina Steenbeek, financial reporting grid business developer at ABN AMRO, described the problems of multiple data silos and complicated app landscapes, while Roberts talked about the need for good data governance across an organisation to support data lineage and the need to centralise data. Steenbeek commented: “When a company realises that dramatic change is required, there are always challenges.”
Cultural challenges are also a problem, but can be managed by getting senior executives onboard and demonstrating the impact of data lineage to users by giving them a visual view of data that couldn’t previously be seen. Data related roles, such as data steward, are also needed, along with awareness across all business areas.
Looking at the need to automate data lineage, Nelson commented: “Automation is vital to data lineage, without it you won’t get far. At scale, manual lineage is impossible. You need to instrument the technology environment to create automated lineage.”
With a successful automated solution in place, the benefits of data lineage can be significant. Gentile concluded: “Data lineage is the most opportunistic thing you can do for your organisation. It is pivotal from a data quality perspective and will certainly provide a return on investment.”
Subscribe to our newsletter
