About a-team Marketing Services
The knowledge platform for the financial technology industry
The knowledge platform for the financial technology industry

A-Team Insight Blogs

Databricks Extends Capabilities of Lakehouse Data and AI Platform

Subscribe to our newsletter

Databricks, provider of the Lakehouse data and AI platform, has extended the platform’s capabilities with the addition of advanced data warehousing and governance, data sharing innovations including an analytics marketplace and data clean rooms for data collaboration, automatic cost optimisation for ETL operations, and machine learning (ML) lifecycle improvements.

The company, founded by the creators of open source solutions Delta Lake, Apache Spark and MLflow, works across business sectors including financial services, where its customer base includes the likes of Nasdaq, ABN Amro, Schroders, FIS, and Swedbank.

“Our customers want to be able to do business intelligence, AI and machine learning on one platform, where their data already resides. Databricks Lakehouse Platform gives data teams all of this on a simple, open, and multi-cloud platform,” says Ali Ghodsi, co-founder and CEO at Databricks.

The company’s additional data warehousing capabilities include Databricks SQL Serverless, available in preview on AWS and providing fully managed elastic compute for improved performance at a lower cost; Photon, a query engine for lakehouse systems that will be made generally available on Databricks Workspaces in coming weeks; open source connectors for Go, Node.js, and Python, to make it simpler to access the lakehouse from operational applications; and Databricks SQL CLI, enabling developers and analysts to run queries directly from their local computers.

Data governance additions include Unity Catalog, which will be made generally available on AWS and Azure, and provides centralised governance for all data and AI assets, with built-in search and discovery, and automated lineage for all workloads.

The company’s marketplace for data and AI will be available later this year, providing a place to package and distribute data and analytics assets. Unlike pure data marketplaces, Databricks’ offering enables data providers to package and monetise assets such as data tables, files, machine learning models, notebooks and analytics dashboards. Cleanrooms, also available later this year, will provide a way to share and join data across organisations with a secure, hosted environment and no data replication required.

ML advancements include MLflow 2.0, which includes MLflow Pipelines that can handle the operational set up of ML for users. Instead of setting up orchestration of notebooks, users can define the elements of the pipeline in a configuration file and MLflow Pipelines manages execution automatically. Beyond MLflow, Databricks has added serverless model endpoints to directly support production model hosting, as well as model monitoring dashboards to analyse real-world model performance.

Delta Live Tables is an ETL framework using a simple, declarative approach to building data pipelines. Since its introduction earlier this year, Databricks has expanded the framework with a new performance optimisation layer designed to speed up execution and reduce the costs of ETL.

Ghodsi concludes: “These new capabilities are advancing our Lakehouse vision to make it faster and easier than ever before to maximise the value of data, both within and across companies.”

Subscribe to our newsletter

Related content


Upcoming Webinar: Approaches to ESG data for analytics

Date: 29 September 2022 Time: 10:00am ET / 3:00pm London / 4:00pm CET Duration: 50 minutes Volumes of ESG data are huge and continue to grow, questioning how financial institutions with a focus on ESG investing can continuously capture and contain required data sets, master and integrate the data, and ensure data quality for meaningful...


SmartStream Combines Air, RDU to Offer Comprehensive Transaction Reporting for MiFID II

SmartStream Technologies has combined its SmartStream RDU reference data utility with its SmartStream Air (Artificial Intelligence Reconciliations) solution to create a new offering that provides regulated entities with comprehensive reporting capabilities for MiFID II. The new solution – Transaction Reporting Reconciliation and Reporting Decision Control – aims to address regulators’ growing requirement for completeness and accuracy...


RegTech Summit APAC

Now in its 2nd year, the RegTech Summit APAC will bring together the regtech ecosystem to explore how capital markets in the APAC region can leverage technology to drive innovation, cut costs and support regulatory change. With more opportunities than ever before for RegTech to add value, now is the time to invest for the future. Join us to hear from leading RegTech practitioners and innovators who will share insights into how they are tackling the challenges of adopting and implementing regtech and how to advance your RegTech strategy.


ESG Data Handbook 2022

The ESG landscape is changing faster than anyone could have imagined even five years ago. With tens of trillions of dollars expected to have been committed to sustainable assets by the end of the decade, it’s never been more important for financial institutions of all sizes to stay abreast of changes in the ESG data...