The leading knowledge platform for the financial technology industry
The leading knowledge platform for the financial technology industry

A-Team Insight Blogs

Tech Matters with Pete Harris: AWS Sharpens Analytics and Big Data Chops

Amazon’s AWS (for Amazon Web Services) cloud platform has been around for a few years now – it was launched in 2006 by the online retailer then looking to make use of its spare IT capacity – and has been increasingly used by financial markets players looking for less expensive compute and storage resource. Recently, however, it has added a number of offerings that could make it very attractive to those in our space looking to manage big data sets and crunch analytics. It’s also begun to showcase third party solutions relevant to the financial services community.

For those new to AWS, it’s basically a way to tap into compute and storage in the cloud on a pay-per-use model. This can be attractive for companies both small and large. For smaller companies, it is possible to rent virtually as much compute power as needed (and only when needed) without investing in the infrastructure in house. Many smaller hedge funds and proprietary traders have taken advantage of this opportunity to run analytics and back testing, which are usually both compute and data intensive tasks that only need to be run perhaps once per day.

For larger companies, the cloud approach can save much time to market for individual business units, since deploying servers for line-of-business applications in an in-house data centre can often be a tedious and bureaucratic process – involving centralised internal IT groups often with many competing priorities – and one resulting in hefty internal charge backs. By contrast, AWS resources are available within minutes, with costs charged to a credit card.

Xignite is one company that has tapped AWS for its enterprise market data offering from the get go. As well as providing a number of public datasets via a web services interface, the company has since moved into offering private services for market participants, allowing them to also distribute internal data across their enterprises.

In fact, data management applications are increasingly being written with cloud deployment in mind, whether it be Hadoop implementations from the likes of Cloudera, to in-memory fabrics from ScaleOut Software.

AWS began life with a couple of core services: EC2 (Elastic Compute Cloud) providing compute power; and S3 (Simple Storage Service) offering block storage. Various options for those exist today, including a choice of Microsoft and Linux operating systems, and the ability to choose shared or dedicated instances (Amazon’s name for a virtual server).

Over the years, and especially in the past few months, AWS has rolled out a number of additional offerings, several with a view to management of big data and running of high performance computing (HPC) workloads. These include:

  • Instances tuned for HPC workloads, with up to 32 processor cores for parallelisation, and GPU co-processors for floating point computation.
  • Instances designed for in-memory and data-intensive applications, with solid state disks.
  • Elastic Block Store persistent storage for applications, so data is retained when applications are no longer actively running.
  • A number of hosted data stores, such as Relational Database Service (RDS), DynamoDB (NoSQL) and ElastiCache (in-memory).
  • Redshift, a petabyte scale data warehouse, based on technology from ParAccel (acquired by Actian).
  • Elastic MapReduce to offer Hadoop functionality in the cloud.
  • Kinesis, providing event stream processing at scale for real-time analytics on so-called big data in motion.

Taken together, this set of value added offerings provide many of the features found in enterprise middleware stacks from the likes of IBM, SAP, Software AG and Tibco Software – companies that are well known in the financial markets space. As such, AWS is increasingly well positioned to benefit as financial enterprises look to leverage cloud services.

Making AWS more attractive for the financial markets is the recently launched financial services category in the Amazon Marketplace – a directory of applications that have been designed for, or ported to, AWS. Instances can be pre-configured with these applications so they are essentially turnkey to deploy.

Kensho Technologies is one recently launched company offering quantitative analysis tools based on AWS technology. In fact, Kensho is working with Nasdaq OMX to make use of FinQloud, the latter’s secure AWS-based cloud.

Other financial applications include: ChartIQ for financial visualisation; CloudPrime for trade confirmations; and Digital Reasoning’s machine learning analytics. And to keep all the instances in AWS in synch with one another, FSMLabs recently released its TimeKeeper product for the cloud.

For sure, AWS isn’t the only cloud game in town – there’s Azure from Microsoft, IBM’s SoftLayer offering, the Google Cloud Platform, services from Rackspace, and new developments from the EMC/VMware Pivotal spinout – what AWS provides today is probably ahead of its competition in terms of completeness of functionality.

While AWS isn’t likely to be seen in low-latency trading environments – offerings from the likes of the recently emerged Lucera are more geared to those extremes – as analytics on big data becomes more of the trading equation, so building and maintaining fast-to-deploy and cost effective infrastructure will become more important. And banks don’t really want to be in that business.

Pete Harris is Principal of Lighthouse Partners, an Austin, TX-based consulting company that helps innovative technology companies with their marketing endeavors. www.lighthouse-partners.com.

Related content

WEBINAR

Recorded Webinar: Market data in the cloud

The Covid-19 pandemic has created new demand for financial information delivery infrastructure to accommodate the many trading and support personnel now working from home (WFH). For many firms, new cloud delivery and hosting capabilities offer a viable solution for supporting these staff, accelerating demand for cloud-based market data delivery infrastructures. This development has thrown up...

BLOG

The Deal is Done – LSEG Completes Acquisition of Refinitiv

A big day in the City last Friday as the London Stock Exchange Group (LSEG) completed its all-share acquisition of Refinitiv first mooted back in July 2019. The acquisition is expected to create a leading, UK-headquartered, global financial market infrastructure provider with a strong data and analytics business, significant capital market capabilities across multiple asset...

EVENT

TradingTech Summit Virtual

TradingTech Summit (TTS) Virtual will look at how trading technology operations can capitalise on recent disruption and leverage technology to find efficiencies in the new normal environment. The crisis has highlighted that the future is digital and cloud based, and the ability to innovate faster and at scale has become critical. As we move into recovery and ‘business as usual’, what changes and technology innovations should the industry adopt to simplify operations and to support speed, agility and flexibility in trading operations.

GUIDE

Regulatory Data Handbook – Third Edition

Need to know all the essentials about the regulations impacting data management? Welcome to the third edition of our A-Team Regulatory Data Handbook which provides all the essentials about regulations impacting data management. A-Team’s series of Regulatory Data Handbooks are a great way to see at-a-glance: All the regulations that are impacting data management today...