Measuring time with great accuracy – and ensuring it is consistent across the many servers that make up a trading enterprise – has historically been costly and difficult. IntelligentTradingTechnology.com talked to Victor Yodaiken, CEO of FSMLabs, to find out how his company is making it easier.
Q: What does FSMLabs – and your TimeKeeper software – focus on?
A: TimeKeeper is an enterprise solution to time synchronisation – it’s aimed not only at getting precisely synchronised time to applications but at eliminating most of the engineering/IT effort needed to architect and administer fault-tolerant high precision time distribution in networks. We’ve also added some hardware – Network Time Servers so that we have a complete end-to-end solution. FSMLabs is a system development company that focuses on markets where there is a compelling business need for higher performance than existing solutions provide. In the time synchronisation market our software is displacing free software that does not work all that well and that absorbs a lot of engineering time – we’re aiming at customers who really need it to work out of the box and to have the performance advantages and robustness that we provide.
Q: Why is it important to synchronise time in distributed systems? Why is it need for trading applications? And other applications?
A: Transactions on the exchanges and within proprietary platforms are so fast that trading platforms can become completely confused about the meaning of the data they are processing if their idea of the time is slightly incorrect. This is especially true for low latency traders but not just for them. Consider a firm that has a co-located trading system in Chicago and one in New Jersey and prices of asset X in Chicago always go up just before prices of Y in New Jersey always go down. If the Chicago system time is off – maybe even by a few microseconds – the system may miss the correlation or, worse, get it backwards. TimeKeeper keeps the system times tightly coupled, compensates for bad time feeds when it can and complains when it cannot. There are many many ways for time feeds to go bad and TimeKeeper is unique in its ability to detect such problems and fail-over robustly. We have had multiple customers who were surprised to find during TimeKeeper evaluations that their existing time feeds were erratic or not working at all. For example, standard free software time clients will accept seconds or even multiple hours jumps in time feeds, even backward jumps, without protest or notice. Having time silently jump backward by a couple of seconds at the wrong time in the trading day is something our customers want to be protected from.
As trading firms make more use of co-location, distributed processing, and cloud computing (TimeKeeper does really well in virtual machines too), the need for precise time synchronisation will only grow. Precise time synchronisation has much wider application than financial trading. Oracle distributed databases need synchronised time, Google has built huge world-wide data storage that relies on time synchronisation to reduce the costs of keeping data consistent. Old solutions do so poorly in virtual machines that they can make systems fail when ported to the cloud. Precise timing makes diagnosis systems that use logs work more efficiently (or at all) because logs of operations on distributed platforms all refer to the same time base. And then there are applications outside the enterprise – like managing electric power distribution.
Q: What sources of time does TimeKeeper work with – and how accurate are they?
A: TimeKeeper software will take time from either of the major network time distribution protocols, PTP 1588 and NTP, and from most other time sources, such as GPS clocks, IRIG feeds, pulse-per-second feeds. One of the advantages of TimeKeeper GrandMaster hardware is that it is native 10Gbps which means it can deliver time to TimeKeeper client software (and any other client software) with minimal delay and error in the high speed networks that are standard in financial trading. Time sources with lower speed connectivity introduce errors – think of a messenger who has to take a couple of local trains before getting on the express. That said, TimeKeeper will do the best it can with any source and on decent networks can keep time well within a microsecond of the reference time.
Our NTP implementation is good enough to match PTP and even do better in some environments. This is really critical in reducing the cost and disruption of upgrades to existing networks which are often built around older NTP software. We improve NTP performance and let customers mix and match.
Q: How does TimeKeeper work with network switches and interface cards to distribute precision time? And how does it fit in with latency monitoring systems, from the likes of Corvil and TS-Associates?
A: Many modern network cards have onboard timestamp hardware that is automatically used by TimeKeeper to compensate for operating system and network stack time delays. We’re starting to see network devices that will timestamp packets too – this allows TimeKeeper another way of compensating for network delays. Our algorithms are sophisticated and effective even without hardware support but we take advantage of what’s there.
The latency monitoring systems from companies like Corvil are intended to solve a different problem than TimeKeeper does. We are concerned with getting precise time to trading applications as efficiently as possible. Those system are concerned with monitoring timing at the network level – and with modern computing and trading software, the network level is a long way from the application.
TimeKeeper is aimed at putting time intelligence in the application. That is, the application can send a message from, say, Chicago to New York with some timestamped information and the New York site can read that message, calculate how fresh it is by comparing the time it was sent in Chicago to the current time in New York, and make an immediate decision. TimeKeeper does give applications insight into latency – application/application latency, not network device to network device.
Q: You recently introduced an appliance to augment your software product. What is it, and why did you decide to develop it?
A: We wanted something that would enable a more seamless end-to-end solution at lower cost. Our GrandMaster appliances fit the high end server/networking environment of financial trading: faster and more flexible networking, better management and integration with reporting/management features, standard server robustness (like dual power supplies), and archival storage of logs. They also have a very high quality oscillator, cost less than competing products, and have faster delivery time. Much of this is due to our software base which allows us to piggyback on the improvements in price/performance of commodity hardware.
We were finding that TimeKeeper software deployments were being delayed by difficulties in acquisition, deployment, and management of time appliances. As mentioned earlier, native 10Gbs is a big advantage in a high end transaction system.
Q: You partner with companies like Symmetricom and Spectracom, which on the face of it would seem to be competitors. How does that work?
A: TimeKeeper works well with and adds value to the solutions from those companies and the market is growing rapidly so it’s not like we have to fight for space. Also, our focus is on the software while those companies are hardware focused. Many of our customers use Spectracom and Symmetricom clocks, antennas, and cards. TimeKeeper works with everything.
Q: What business drivers in the financial markets do you see shaping the future of time synchronisation, and your product focus?
A: Precise time synchronisation in financial trading was a cutting edge technology just a few years ago: now it’s becoming something that has to be available everywhere, has to be robust, and has to produce an auditable trace both for regulatory reasons and to demonstrate due diligence. What we see is that the jury-rigged “science project” solutions that worked for some organisations a few years ago are being recognised as too risky and costly.
Our growth is being driven by business units that want to be able to focus on their trading or other business with time synchronisation as a solved problem that does not produce surprises or involve costly engineering interventions. There is no competing product that solves these problems nearly as well or as comprehensively as we do, so we think we have a couple of years of being in a very strong position.
And we have a lot of new technology on the way that will increase the value of our products in emerging very high speed and software defined networks, technologies that add latency intelligence to applications and improve task scheduling and performance. It turns out that if you have the data for time distribution, you can find many network problems like slow or misconfigured switches and routers, overloaded servers or applications, and poorly performing network adaptors. We should be announcing new products in early fall 2013.
Subscribe to our newsletter
