About a-team Marketing Services
The knowledge platform for the financial technology industry
The knowledge platform for the financial technology industry

A-Team Insight Blogs

ScaleOut Pushes Hadoop Towards Low-Latency for Real-Time Analytics

Subscribe to our newsletter

OK, so the headline is a tad extreme, but bear with me. Recent developments combining in-memory technologies and Hadoop/MapReduce from ScaleOut Software point to a future where big data analytics and real-time processing, as it’s defined in the financial markets, could meet.

ScaleOut has just released its ScaleOut hServer V2, an in-memory data grid, which it claims can boost Hadoop performance by 20x, and can make it suitable for processing ‘live data’ to deliver ‘rea-ltime analytics’.

“To minimise execution time, ScaleOut hServer employs numerous optimisations to minimise data motion during the execution of MapReduce applications, and it can automatically cache HDFS data sets within the IMDG (a feature introduced with ScaleOut hServer V1). In addition, ScaleOut hServer’s memory capacity and throughput can be scaled by adding servers to the IMDG’s cluster. The product automatically rebalances the data set and execution workload when servers are added or removed,” says the company in a statement.

As well as boosting performance of a Hadoop deployment, hServer also incorporates Map/Reduce logic so that a Hadoop distribution is not actually required – though the company suggests its offering is not a direct replacement for Hadoop.

Nevertheless, “ScaleOut hServer is designed to be compatible with most Java-based Hadoop Map/Reduce applications developed for the standard Hadoop distributions, requiring only a one-line code change to execute applications using ScaleOut hServer.”

The big picture here is that ScaleOut – as well as other companies pushing in-memory technology – is recognising that the batch-oriented nature of Hadoop has limitations for real-time applications, such as those found in the financial markets.

While ScaleOut is today looking to boost Hadoop performance to make applications that used to take hours and minutes to execute run now in minutes and seconds, the performance trajectory could well follow that of the low-latency space, where milliseconds gave way to microseconds, and now nanoseconds.

The deployment of multi-core and multi-socket servers, GPU technologies and advances in memory will all benefit data grid vendors like ScaleOut, as well as Hadoop and other big data analytics offerings.

Subscribe to our newsletter

Related content

WEBINAR

Upcoming Webinar: Enhancing Buy-Side Trading Efficiency: Navigating Interoperability and AI in Real Workflows

Date: 26 June 2024 Time: 10:00am ET / 3:00pm London / 4:00pm CET Duration: 50 minutes Enhancing Buy-Side Trading Efficiency: Navigating Interoperability and AI in Real Workflows Emerging capabilities in AI and interoperability are transforming trading workflows, with the promise of heightened levels of collaboration and personalisation resulting in greater efficiency and performance. The potential...

BLOG

Devexperts Enhances Multi-Currency Support in DXtrade XT

Devexperts, the software solutions provider for capital markets, has unveiled a significant new version of its flagship platform, DXtrade XT, which introduces extensive multi-currency support, enabling brokers to expand their trading capabilities across international markets. In order to address the increasing demand from brokers for platforms that facilitate multi-market trading, DXtrade XT – initially launched...

EVENT

ESG Data & Tech Summit London

The ESG Data & Tech Summit will explore challenges around assembling and evaluating ESG data for reporting and the impact of regulatory measures and industry collaboration on transparency and standardisation efforts. Expert speakers will address how the evolving market infrastructure is developing and the role of new technologies and alternative data in improving insight and filling data gaps.

GUIDE

Impact of Derivatives on Reference Data Management

They may be complex and burdened with a bad reputation at the moment, but derivatives are here to stay. Although Bank for International Settlements figures indicate that derivatives trading is down for the first time in 10 years, the asset class has been strongly defended by the banking and brokerage community over the last few...