Chicago-based SpryWare recently announced its ticker plant can process 10 million messages per second. IntelligentTradingTechnology.com talked to the company’s co-founder Daniel May about the market data explosion, how SpryWare is responding, and trends for 2011.
Q: The latest capacity test on your MIS ticker plant show that it can process 10 million messages per second. Is that going to be enough to keep up with the global markets in 2011?
A: We use the information from the Financial Information Forum’s Market Data Capacity Working Group, which covers most but not all the exchanges. The peak for November 2010 was 6,293,042 messages per second, but that number is only an estimate, simply summing the peaks across all reported exchanges.
Since the peaks rarely occur at the same time of the day, this would be a worst case scenario. If you take the same approach and add in the May 2011 projections, the number becomes 7,179,107 mps. This is a projected increase of 14%. Now, for a real world US number, take the peak of 3,787,296 msg/sec at 10:00am on November 18, 2010 from marketdatapeaks.com and forecast out by the 14% projected six months growth. All these numbers are manageable under the 10 million mps we can handle. Keep in mind that our capacity is measured on one server. We can always add more.
Q: Can you cite some of the key aspects of market data handling that impact processing speed?
A: Certain feeds take longer (more cycles) to process a message than others. Things like decompression, symbol resolution, order book insertion/deletion all come into play. In general, top of book feeds can be processed faster than depth or order book feeds, and binary feeds faster than ASCII feeds. It all boils down to how much “work” you need to do on a market data message to normalise it and get it ready to hand off to a client, such as an algo trading app).
Q: When you say ‘process’ a message, what is included in that processing?
A: Good question, in this case, it means reading it off the wire from the network interface card, decompressing (if necessary), resolving implied data (for example, the symbol may be a binary number that needs to be looked up from a table sent out pre trading), normalising the data to the SpryWare Market Data Message, updating the various local cache databases (top of book, order books, aggregated order books etc), and publishing the message back on the wire to interested subscribers.
Q: You mention that your test setup includes Intel microprocessors, network cards from Solarflare Communications, and the Linux operating system from Red Hat. How do each of these help to produce this high performance? What does each bring to the party?
A: They all contribute, some more than others. We are purposely not being specific about where the major gains are made, other than the software optimisations of the v3 MIS ticker plant code was the biggest improvement. The other three combined together offer more overall CPU to the application to process more messages per second.
Q: More generally, what market data delivery trends do you see playing out in 2011?
A: I think for the high-frequency, low-latency crowd, you are going to see more in-process, on the box movement and less emphasis on off the box middleware deployments. For the not so latency sensitive, it’s more about capacity and reducing overall hardware footprint and operational costs. Of course, new content will always be an upfront topic.
Q: And where will SpryWare be focusing its development effort? Is it just about performance? Or will there be functionality enhancements too?
A: Both, and they always come hand in hand. Reducing latency is always ongoing, and that usually results in higher capacity. We continue to add business logic that helps speed deployment of trading applications like smart books, value added calculations, and more robust time series offerings.
Subscribe to our newsletter
