The leading knowledge platform for the financial technology industry
The leading knowledge platform for the financial technology industry

A-Team Insight Blogs

Tech Matters with Pete Harris: AWS Sharpens Analytics and Big Data Chops

Amazon’s AWS (for Amazon Web Services) cloud platform has been around for a few years now – it was launched in 2006 by the online retailer then looking to make use of its spare IT capacity – and has been increasingly used by financial markets players looking for less expensive compute and storage resource. Recently, however, it has added a number of offerings that could make it very attractive to those in our space looking to manage big data sets and crunch analytics. It’s also begun to showcase third party solutions relevant to the financial services community.

For those new to AWS, it’s basically a way to tap into compute and storage in the cloud on a pay-per-use model. This can be attractive for companies both small and large. For smaller companies, it is possible to rent virtually as much compute power as needed (and only when needed) without investing in the infrastructure in house. Many smaller hedge funds and proprietary traders have taken advantage of this opportunity to run analytics and back testing, which are usually both compute and data intensive tasks that only need to be run perhaps once per day.

For larger companies, the cloud approach can save much time to market for individual business units, since deploying servers for line-of-business applications in an in-house data centre can often be a tedious and bureaucratic process – involving centralised internal IT groups often with many competing priorities – and one resulting in hefty internal charge backs. By contrast, AWS resources are available within minutes, with costs charged to a credit card.

Xignite is one company that has tapped AWS for its enterprise market data offering from the get go. As well as providing a number of public datasets via a web services interface, the company has since moved into offering private services for market participants, allowing them to also distribute internal data across their enterprises.

In fact, data management applications are increasingly being written with cloud deployment in mind, whether it be Hadoop implementations from the likes of Cloudera, to in-memory fabrics from ScaleOut Software.

AWS began life with a couple of core services: EC2 (Elastic Compute Cloud) providing compute power; and S3 (Simple Storage Service) offering block storage. Various options for those exist today, including a choice of Microsoft and Linux operating systems, and the ability to choose shared or dedicated instances (Amazon’s name for a virtual server).

Over the years, and especially in the past few months, AWS has rolled out a number of additional offerings, several with a view to management of big data and running of high performance computing (HPC) workloads. These include:

  • Instances tuned for HPC workloads, with up to 32 processor cores for parallelisation, and GPU co-processors for floating point computation.
  • Instances designed for in-memory and data-intensive applications, with solid state disks.
  • Elastic Block Store persistent storage for applications, so data is retained when applications are no longer actively running.
  • A number of hosted data stores, such as Relational Database Service (RDS), DynamoDB (NoSQL) and ElastiCache (in-memory).
  • Redshift, a petabyte scale data warehouse, based on technology from ParAccel (acquired by Actian).
  • Elastic MapReduce to offer Hadoop functionality in the cloud.
  • Kinesis, providing event stream processing at scale for real-time analytics on so-called big data in motion.

Taken together, this set of value added offerings provide many of the features found in enterprise middleware stacks from the likes of IBM, SAP, Software AG and Tibco Software – companies that are well known in the financial markets space. As such, AWS is increasingly well positioned to benefit as financial enterprises look to leverage cloud services.

Making AWS more attractive for the financial markets is the recently launched financial services category in the Amazon Marketplace – a directory of applications that have been designed for, or ported to, AWS. Instances can be pre-configured with these applications so they are essentially turnkey to deploy.

Kensho Technologies is one recently launched company offering quantitative analysis tools based on AWS technology. In fact, Kensho is working with Nasdaq OMX to make use of FinQloud, the latter’s secure AWS-based cloud.

Other financial applications include: ChartIQ for financial visualisation; CloudPrime for trade confirmations; and Digital Reasoning’s machine learning analytics. And to keep all the instances in AWS in synch with one another, FSMLabs recently released its TimeKeeper product for the cloud.

For sure, AWS isn’t the only cloud game in town – there’s Azure from Microsoft, IBM’s SoftLayer offering, the Google Cloud Platform, services from Rackspace, and new developments from the EMC/VMware Pivotal spinout – what AWS provides today is probably ahead of its competition in terms of completeness of functionality.

While AWS isn’t likely to be seen in low-latency trading environments – offerings from the likes of the recently emerged Lucera are more geared to those extremes – as analytics on big data becomes more of the trading equation, so building and maintaining fast-to-deploy and cost effective infrastructure will become more important. And banks don’t really want to be in that business.

Pete Harris is Principal of Lighthouse Partners, an Austin, TX-based consulting company that helps innovative technology companies with their marketing endeavors.

Related content


Upcoming Webinar: Infrastructure monitoring: mapping technical performance to business performance

Date: 8 July 2021 Time: 10:00am ET / 3:00pm London / 4:00pm CET Duration: 50 minutes It’s a widely recognized truth that if you can’t measure something, you can’t improve its performance. As high-performance connectivity technologies have established themselves in the mainstream of financial firms’ trading architectures, the ability to monitor messaging and data infrastructures...


Itiviti Seeks First-Mover Advantage through Shift to Cloud

Nordic Capital-owned trading technology provider Itiviti is seeking first-mover advantage from its plans to make its entire electronic trading platform available through the cloud. The company last month announced a collaboration with data specialist Data Canopy and tech consultant Ingram Micro to adopt a hybrid private-public cloud strategy for its suite of trading technology services,...


TradingTech Summit London

The TradingTech Summit in London brings together European senior-level decision makers in trading technology, electronic execution and trading architecture to discuss how firms can use high performance technologies to optimise trading in the new regulatory environment.


Regulatory Data Handbook – Third Edition

Need to know all the essentials about the regulations impacting data management? Welcome to the third edition of our A-Team Regulatory Data Handbook which provides all the essentials about regulations impacting data management. A-Team’s series of Regulatory Data Handbooks are a great way to see at-a-glance: All the regulations that are impacting data management today...