The leading knowledge platform for the financial technology industry
The leading knowledge platform for the financial technology industry

A-Team Insight Blogs

Using Data Lineage to Manage Regulatory Compliance and Financial Reporting

By Mark Hermeling, Chief Technology Officer at Asset Control.

Within data management, data lineage is increasingly becoming an important capability. From pinpointing the origins of data to how it evolves over time allows a wide range of sectors to achieve a clear view of their data, track the data exactly, and use it to assist with managing regulatory compliance and financial reporting.

Increasingly, data lineage is becoming hard-wired into regulations and data quality frameworks – ultimately the need is for ‘explainability’. If a bank, for example, values a position at $25 million, it might need to explain why it is valued at that amount, how it came to that decision and what data points it used in arriving at the valuation. All this context, and more, may need to be tracked.

Corporate treasurers and finance staff have a major role to play in ensuring that the process of reporting and consolidating financial data is consistently transparent and explainable. They can also use data lineage and the related concept of explainability in other contexts. When changes are made to existing processes, for example, data lineage can be used in diagnostics to improve data quality. It can also be key to data licensing. If an organisation is licensing third-party content and therefore has to abide by associated restrictions, it will need to know what data is a derivative of another piece of data.

Finally, data lineage can be key to achieving better management of client records and ensuring greater care is taken about where client data is moved to and used going forward. After all, if a business client under GDPR demands that their data be expunged from the records, the organisation will need to know where all of that customer’s details have ended up in order to be able to achieve this.

Horizontal and vertical lineage

To do all this efficiently and well, organisations will effectively need to implement two different kinds of data lineage: horizontal and vertical. Horizontal data lineage traces the journey of a piece of data as it moves through systems from source to destination. It effectively tracks the journey of a specific item of data – typically across systems and reports – and is focused on the process of keeping track of the data that the business has consumed, where it subsequently went, and who touched it on its journey. The objective is to trace the journey of the data upstream.

Vertical data lineage, in contrast, describes the transformations that happen to a piece of data on the journey. The data could be an element feeding into a calculation, one of the sources of a bond curve calculation, for example. The lineage in this case would be to ‘go back’ from the bond curve and see what individual bonds formed part of the input at a specific point in time.

In short, horizontal data lineage traces data back to the original source, while vertical data lineage reverse-engineers the transformations that happen along the way, whether they are simple processes like cross-referencing, or tracking the different taxonomies that exist for financial instruments or industry classifications. Often, in order to compare like for like, an organisation might want to express an issuer or counterparty within the same taxonomy. So, for example, if one taxonomy labels a segment ‘IT’ and another calls the same segment ‘computer systems’, the organisation may need to ensure that the same label is used for both.

Meeting the Challenge

The specific challenge in terms of an organisation’s ability to reverse engineer is that it will need to keep track of input data sources and their value at the time a transformation took place, all the calculation parameters that fed into the calculation and their value at the time it was done, and the algorithm that was used.

To address the data lineage challenge, firms need bi-temporality so they know the value of the data and associated business logic at the time the calculation took place. They need to be able to track metadata and keep cross-reference tables between different taxonomies and classification schemes up to date, and they need a clear administrative process detailing who can access data, where it goes, where did they get it from, and what their sources were. Moreover, they also need a sourcing hierarchy to be able to look at data sources that are clearly documented and provide access for everybody who needs the data.

This is a complex undertaking, but organisations, and more specifically treasury departments, that understand the requirement and can put the right combination of processes and technology in place will be best placed both to meet data lineage requirements and gain competitive edge.

Related content


Upcoming Webinar: Sourcing and managing ESG disclosure data to ensure compliance

Date: 8 February 2022 Time: 10:00am ET / 3:00pm London / 4:00pm CET Duration: 50 minutes As interest in ESG investing continues to accelerate, asset managers must source and manage increasing volumes of ESG disclosure data. Data volumes are not the only problem, with different types of ESG disclosure data, the need to master the...


How Knowledge Graphs Can Deliver Insights When Integrating Decentralised Data

The majority of large data integration projects falter because systems are developed in a bespoke fashion, leaving organisations with critical applications that have been modified and transformed into a hornet’s nest of code that they cannot unravel for fear of breaking something. So, instead of trying to unravel systems, organisations have tended to duplicate processes,...


ESG Regulation, Reporting & Data Management Summit (Redirected)

This summit will explore challenges around assembling and evaluating ESG data for reporting and the impact of regulatory measures and industry collaboration on transparency and standardisation efforts. Expert speakers will address how the evolving market infrastructure is developing and the role of new technologies and alternative data in improving insight and filling data gaps.


Regulatory Data Handbook 2021/2022 – Ninth Edition

Welcome to the ninth edition of A-Team Group’s Regulatory Data Handbook, a publication dedicated to helping you gain a full understanding of regulations related to your organisation from the details of requirements to best practice implementation. This edition of the handbook includes a focus on regulations being rolled out to bring order and standardisation to...