Implementing a high-volume, low-latency market data processing system on commodity hardware using IBM middleware

Abstract

A stock market data processing system that can handle high data volumes at low latencies is critical to market makers. Such systems play a critical role in algorithmic trading, risk analysis, market surveillance, and many other related areas. We show that such a system can be built with generalpurpose middleware and run on commodity hardware. The middleware we use is IBM System S, which has been augmented with transport technology from IBM WebSphere MQ Low Latency Messaging. Using eight commodity x86 blades connected with Ethernet and Infiniband, this system can achieve 80 µsec average latency at 3 times the February 2008 options market data rate and 206 µsec average latency at 15 times the February 2008 rate.

Evaluation of a high‐volume, low‐latency market data processing system implemented with IBM middleware

“…Additional experiments with the platform from our previous paper seem to indicate that the effectiveness of LLM tuning depends, to a great extent, on the system load 8. In our previous paper, we used every optimization except for pinning and LLM tuning.…”

Design principles for developing stream processing applications

“…Several such applications are also prevalent in the financial services front‐office, where increasingly large amounts of market data need to be processed with millisecond latencies to determine trading opportunities 14. Streaming data feeds are obtained from stock exchanges and processed using extremely lightweight and responsive analytics that are dynamically reconfigured at controlled intervals.…”

“…Streaming data feeds are obtained from stock exchanges and processed using extremely lightweight and responsive analytics that are dynamically reconfigured at controlled intervals. An example application involves option market‐making 14. This application must calculate and present option prices to the market and requires a real‐time snapshot of the market data to perform its calculations.…”

“…The availability of live streaming data and more importantly, the ability to process and analyze all of this data as it is streaming, enables more informed decision‐making and extraction of new insights in real time. This directly results in value in many contexts such as customer retention 12, waste reduction 13, profit increase 14, fraud detection 15, and pattern detection, providing the insights needed to leverage new business opportunities. All these can ultimately provide a competitive edge to businesses, increased efficiency to governments, and fuel new scientific discoveries.…”

From a stream of relational queries to distributed stream processing

“…In reality, a lot of applications implemented by companies and research institutions can be construed as data analytics applications. The fundamental shift that is happening over the last few years, however, is the availability of more flexible tools for implementing this breed of applications [11,35,39] such that they can scale to the data volumes and rates that are now becoming common and can provide timely responses.…”