Investment firms have never had access to more data. From corporate filings and social media to satellite imagery, transaction data and other alternative sources, the range of information that can potentially feed the investment process continues to expand.
But as AI becomes embedded more deeply into research and investment workflows, the challenge is shifting. Simply acquiring and storing large volumes of data is no longer enough. Increasingly, firms need to ingest, query and extract value from that information in real time – and to do so without allowing the underlying infrastructure costs to spiral.For Aerospike, a high-performance real-time NoSQL database provider, that is forcing investment firms to reconsider some of the assumptions underpinning their data architecture.
“We have traditionally talked about the three Vs of data: volume, variety and velocity,” says Greg Georges, Senior Solutions Architect at Aerospike in conversation with Market & Alt Data Insight. “In the past, companies could often process that data offline. They might run machine-learning models or other forms of analysis, process the data, derive a result and then use that result 24 or 48 hours later. What we are seeing now is a requirement for much more immediate processing.”
From trading speed to research speed
Real-time data is hardly a new requirement in financial markets. Trading firms have spent decades engineering systems capable of processing market information and responding within increasingly narrow timeframes.
What is changing is the range of workflows to which those expectations now apply. Georges cites financial institutions doing massive scale fraud prevention, and a hedge fund using Aerospike as the foundation for a research environment supporting its advisers – an example of real-time requirements extending beyond execution and into the investment research process.
AI adds another dimension. Investment firms potentially need to work with vast quantities of unstructured information – including PDFs, audio and social media – alongside conventional structured datasets. Sentiment analysis, document analysis and other AI-enabled techniques can make more of that information usable, but only if the underlying infrastructure can supply the relevant data when it is needed.
“We are seeing enormous amounts of data that increasingly need to be taken into account in real time to provide answers and value as quickly as possible,” says Georges. “The challenge ultimately comes down to both real-time processing and the volume of data involved. In the AI environment, data is no longer simply what you find on the internet. It could be prospectuses, other unstructured information, social media posts or data used for sentiment analysis around a particular company.”
The economics of real-time data
Speed, however, is only part of the equation. A database can potentially deliver high throughput and low latency by throwing more computing resources at the problem. For investment firms facing rapidly expanding datasets and computationally intensive AI workloads, the cost of doing so becomes increasingly important.
One of Aerospike’s core value propositions is that its architecture can support large-scale, low-latency workloads with a smaller infrastructure footprint. Georges gives the example of a workload requiring 1,000 virtual machines or nodes on one architecture potentially being reduced to around 150 with Aerospike, although the actual reduction will naturally depend on the workload and environment.
That changes the discussion from whether real-time processing is technically possible to whether it is economically sustainable as data volumes grow.
Making data reusable
There is another issue that is particularly relevant to alternative data: acquiring and storing information does not automatically make it useful.
Firms increasingly need data architectures capable of serving multiple consumers – from analysts and portfolio managers to quantitative models and AI applications. That makes the way information is structured, queried and made available across the organisation increasingly important.
“Capturing and storing the data is one thing. You can put it into object storage or many other repositories,” says Georges. “The more important question is how you model that data in a structure that allows it to be reused by as many parts of the organisation as possible. So it is not simply about the volume of data or how it is captured. It is also about storing and structuring it in a way that maximises reuse across the business.”
That potentially changes the economics of alternative data again. The value of a dataset no longer depends solely on whether a firm can obtain it or discover a signal within it, but also on how efficiently that information can be operationalised across different investment processes.
AI changes the data stack
The emergence of agentic AI could push that requirement further. Aerospike is integrating with emerging AI frameworks and technologies, including Model Context Protocol (MCP), as databases become part of a broader stack connecting incoming data, models, computing infrastructure and AI agents.
Georges characterises the evolution as a progression from predictive AI – including recommendation engines and fraud detection – towards real-time ingestion and analysis, and now agentic systems capable of interacting dynamically with enterprise data.
For financial institutions, that creates a familiar tension. Firms understandably want infrastructure to be proven before deploying it into critical environments. But waiting also carries a potential cost if competitors can exploit new data and AI capabilities more quickly.
“AI is developing much faster than previous technology shifts such as cloud or mobile,” says Georges. “The difficulty is that firms can also see competitors moving faster. As a result, they may need to make decisions – and experiment with new ways of doing things – in a much shorter timeframe than they have traditionally been comfortable with.”
For alternative data users, the implication is that competitive advantage may increasingly depend on more than finding the right dataset. As AI expands both the volume of information firms can exploit and the speed at which they expect to use it, the architecture sitting underneath that data is becoming part of the investment equation itself.
Greg Georges will discuss this topic in more detail at the A-Team Group/Eagle Alpha Alternative Data Conference in New York on 10th September.
Subscribe to our newsletter
