The leading knowledge platform for the financial technology industry
The leading knowledge platform for the financial technology industry

A-Team Insight Blogs

STAC Benchmarks IBM’s Hadoop

STAC – aka the Securities Technology Analysis Center – has benchmarked IBM’s proprietary Platform Symphony implementation of Hadoop MapReduce, versus the standard open source offering, to compare their respective performance. On average, IBM’s implementation performed jobs 7.3 times faster than the standard, reducing total processing time by a factor of six.

Better known for its benchmarking of low-latency trading platforms, STAC leveraged the Statistical Workload Injector for MapReduce (SWIM), developed by the University of California at Berkeley. SWIM provides a large set of diverse MapReduce jobs based on production Hadoop traces obtained from Facebook, along with information to enable characterisation of each job. STAC says it undertook the benchmarking because many financial markets firms are deploying Hadoop.

The hardware environment for the testbed consisted of 17 IBM compute servers and one master server communicating over gigabit Ethernet. STAC compared Hadoop version 1.0.1 to Symphony version 5.2. Both systems ran Red Hat Linux and used largely default configurations.

IBM attributes the superior performance of its offering in part to its scheduling speed. IBM’s Hadoop is API-compatible with the open source offering but runs on the Symphony grid middleware that became IBM’s with its aquisition of Platform Computing, which closed in January of this year.

For more information on STAC’s IBM Hadoop benchmark, see here.

Related content

WEBINAR

Recorded Webinar: The evolution of market surveillance across sell-side and buy-side firms

Market surveillance is crucial, and in many cases a regulatory requirement, to ensuring orderly securities markets and sustaining confidence in trading. It can be breached and has become increasingly complex in the wake of the Covid pandemic, Brexit, and the emergence of new asset classes. This webinar will review the extent of market abuse in...

BLOG

CJC Extends Availability of mosaicOA to BT Radianz Cloud

Market data technology specialist CJC has extended availability of its mosaicOA infrastructure performance monitoring solution to BT Radianz Cloud, making the IT visualisation and analytics tool the first to be made available to Radianz users. The move follows the recently addition of mosaicOA to Google Cloud. The mosaicOA platform provides firms with interactive dashboards to...

EVENT

Virtual Briefing: ESG Data Management – A Strategic Imperative

This briefing will explore challenges around assembling and evaluating ESG data for reporting and the impact of regulatory measures and industry collaboration on transparency and standardisation efforts. Expert speakers will address how the evolving market infrastructure is developing and the role of new technologies and alternative data in improving insight and filling data gaps.

GUIDE

Data Lineage Handbook 2019

Welcome to our latest handbook on data lineage, a critical concern for data managers working to achieve regulatory compliance, deliver operational gains, and provide meaningful value to the business. The handbook covers the complete scope of data lineage, with a view to helping you win management buy-in and budget, decide whether to build or buy...