As data volumes grow, new sources emerge, and firms migrate to the cloud, data lineage has become essential to mapping, visualising and understanding data across the enterprise, and crucial to providing business users with trusted information. Octopai, which came to market in 2018, offers automated metadata-based data lineage with a focus on analysing and understanding an organisation’s business intelligence (BI) landscape.
The company is based in Rosh Haayin, Israel, raised initial funding in 2016, and spent the next two years developing an innovative product that would be ready when needed. Octopai CEO, Amnon Drori, explains: “Five years ago, we saw a problem that was small but would become large and need a technology solution. The amount of data firms were using was growing rapidly and it was increasingly accessed by business users after it had been though BI and analytics applications – this raises questions about where the data was actually coming from in the data warehouse and could it be trusted?”
Octopai answers these questions – and more – with an enterprise grade Software-as-a-Service (SaaS) automated data lineage platform running on Microsoft Azure. The platform includes machine learning, subtext decision tree analysis, and algos, and collects metadata from every BI tool, regardless of vendor or type, in order to visualise and analyse an organisation’s BI landscape. With automated data lineage in place, business users can trust the data they are using and make better decisions. Drori comments: “90% of the lineage done by Octopai cannot be done manually.”
Solutions supported by the Octopai platform come from traditional data warehouse and BI vendors such as Oracle and Business Objects, as well as from next-generation vendors such as Snowflake, Qlik and Tableau. The platform also includes automated metadata discovery that can map where data fields exist down to code line level in 20 seconds. It is accompanied by an automated business glossary to ensure data consistency across an organisation and ease compliance with regulations such as GDPR that require personally identifiable information (PII) to be located wherever it resides in an organisation.
As financial firms migrate to the cloud and adopt cloud native BI solutions, Octopai is ready with a cloud native version of its platform. The first cloud native solution on the platform is Microsoft Azure Data Factory. With Octopai’s support and analysis of Azure Data Factory, organisations can view end-to-end data lineage from the data factory to reporting. More cloud native applications will be added to the platform as the year progresses.
Drori says: “Evolution doesn’t replace tools, but creates more complex on premise and cloud environments. If BI is on premise, in the cloud or in a hybrid environment, Octopai provides a layer to analyse the complete landscape and provide intelligence on the BI. We can help firms trust their data, make money, save money, and keep their executives out of jail.”
Use cases of automated data lineage described by Octopai include identifying and understanding different answers given by multiple BI tools to the same question; finding the cause of reporting errors by pinpointing the data in question and explaining where it came from and any modifications it went through; supporting system migration and updates; identifying redundant systems and data; ensuring compliance with data privacy regulations; and executing impact analysis of any proposed changes to operational or business processes.
Subscribe to our newsletter
