By Martijn Groot, Vice President, Strategy, Alveo Technology.
In previous decades, people could be forgiven for primarily associating terms such as machine learning and artificial intelligence (AI) with blockbuster films such as The Terminator and The Matrix, but it was, in fact, as long ago as 2017 when Amazon CEO Jeff Bezos described the technology as being in its ‘golden age’. For the financial sector, however, this renaissance has yet to fully take flight.
This is reflected by recent research conducted by Alveo, which found that only 37% of financial services organisations are currently able to incorporate solutions such as AI and machine learning into operational workflows, investment processes and market analysis. Logistical issues commonly await data scientists before they can access data to assist with decision making. With data management and analytics separated, quantitative analysts need to hunt around to gather the data required to develop, train and deploy their models.
The resources and time needed to collect this data means the focus for data scientists is unnecessarily shifted away from their core competency of data analysis. Often, these analysts still find they have to contact the IT department to write a query or set up a report. Even when quants access data, they may find the metadata that should provide insights into the quality and origins of data, and license permissions and approvals, are incomplete or even completely missing. Today, many financial services firms understand they need a better way to provision their data scientists and other key users with clean data sets on financial products and pricing histories.
Big data requires a suitable processing framework. Apache Spark, for example, used in conjunction with cloud-native database technology, allows organisations to move to the next level in scalable, highly performant and integrated data management and analytics. Open source NoSQL database technologies like MongoDB and Cassandra are highly scalable, flexible, and well suited for big data storage and processing.
Some 88% of respondents in the Alveo research reported having made active use of open source technologies in their data management analytics processes, while 36% cited open source as being beneficial in integrating AI and machine learning technologies.
Today, data is often still held in different department level data stores and within legacy applications where accessing it can be complex. The metadata surrounding the data is often not updated frequently, making lineage and understanding the relevant permissions and quality checks it passed difficult.
A lack of clear data catalogues and an increasing number of sources lead to time consuming searches to drive decision making processes. This was identified in the research findings as the top reason data scientists are struggling to find optimum data quality (28%). Using the latest technology to bring data management and analytics together enables them to treat the two as integrated disciplines and helps firms secure fast and flexible access to data with transparency for the preparation process and provenance behind it.
The advantages of integrated data management and analytics
For quants and data scientists, the benefits of integrated data management and analytics are numerous. Some 27% highlighted improved productivity in their daily tasks as the main benefit of integrating market and reference data with advanced data analytics. Better informed decision making requires easy access to clean and consistent data. Metadata information such as usage permissions and quality levels is required to ascertain fitness for purpose. With the help of popular quant languages such as Python and R, they can create a robust and scalable data meeting place, enabling users to share these analytics across their data supply chain and develop a common approach to risk management, performance management, and compliance.
This primarily facilitates increased productivity for quants and data scientists. We are seeing many data analysts today that are looking to dig into the data to find indicators that help them discover investment signals and returns in the market. Increasingly too, they are at least starting to incorporate innovative data science solutions, including AI and machine learning, into market analysis and investment processes.
Adopting this methodology allows proprietary analytics to be built with varying data types, helping to support different aspects of the supply chain and bring together views on the combined data. This approach to user enablement is also helping to democratise analytics, bringing them into the orbit of those who are not data experts.
Future potential
We are entering a new dawn of financial data management. AI and machine learning combined with increased market data provisioning technology are coming together to supply integrated analytics and data management. Financial data subject matter expertise and innovative data management solutions help firms lower the cost of data provisioning, improve decision making, and maximise the mileage they get out of their data.
Subscribe to our newsletter
