Today’s quantitative finance environment is dramatically different from that of just a few years ago. We’ve gone from gigabytes of daily market data that is put in memory for algorithms and jobs isolated to a few machines, to terabytes of data and systems that need to process hundreds of thousands – and even millions – of jobs.
As financial firms seek faster, reliable and more effective order execution that can also help reduce transaction costs, algorithmic trading has experienced tremendous growth. With an expected CAGR (Compound Annual Growth Rate) near 12% through 2026, algorithmic trading will continue to fuel a surge in data and the adoption of data-intensive technologies designed to optimize insights and maximize performance, such as artificial intelligence (AI) and deep learning.
This exponential growth in daily transactions means that algorithmic quant trading firms can no longer store the active trade data sets in memory. Instead, they need to transition to support growing quant teams on increasingly larger data sets – at scale. This creates new performance requirements compared to what was once fairly simple back in the days when infrastructures were very fast, and data could fit in RAM. But the explosion of data and the need to collect, store and analyze more data than ever, has changed – and complicated – environments. Terabytes of data are now generated by smart systems that are more powerful, can handle billions of market transactions, and workloads that perform risk modeling, portfolio management, simulations and other data-driven computations.
From a storage perspective, often SAN and NAS haven’t been able to scale to the volume of data that is being ingested today and, as a result, performance suffers. Without advanced storage, firms would need to reduce the data sets they are working with, give up near real-time performance, or limit the number of simultaneous processes. In today’s competitive market, that’s not a viable option.
Effective quantitative analytics requires the following storage features:
-
- Highly scalable storage to efficiently store billions of files
- High read/write performance
- High user concurrency capability
- Fast interconnect between compute resources and the storage platform
What’s needed? Storage that supports extreme concurrency at scale
High frequency trading and analytics environments are characterized by massive data volumes, low latency and complex processing capabilities. So, it’s not surprising that infrastructure and data modernization have become a priority for quant trading firms. Organizations are moving from infrastructure that supports real time, small data sets to large, highly concurrent workloads with large data sets. Organizations also need to be able to securely support data growth for concurrent workloads to comply with governmental and regulatory standards, even as the number of people simultaneously accessing data grows as well. With financial services experiencing a 1300%+ increase in ransomware attacks in 2021, data security and risk mitigation need to be foundational in today’s quantitative analytics environment, both financially and operationally.
The ideal storage solution enables firms to:
-
- Leverage single copy architecture with a centralized shared storage model. This makes managing a single copy of all market data easier than managing a copy for every compute node.
- Run analysis and AI/DL workloads simultaneously with co-located data that supports both types of jobs and compute. This eliminates data movement or specialized configurations.
- Leverage massive scalability and simplicity by storing data under a single file system and namespace for all applications to use. This provides less operational overhead while allowing you to non-disruptively scale capacity.
- Mitigates risk by increasing resiliency and minimizing vulnerability to cybersecurity threats.
Dell Unstructured Data Solutions for quantitative analytics – Storage that performs, scales and secures
With Dell Unstructured Data Storage (UDS) solutions, financial firms can leverage a portfolio that meets the performance, scalability, and security demands of quantitative analytics environments. In addition to solutions with scale out architecture to enable high bandwidth, high concurrency, and high performance with all flash options, UDS is uniquely suited for enterprise-grade trading:
-
- Leverages efficient single copy architecture for market data
- Runs tick data analytics and AI/DL workloads simultaneously with data in place
- Runs in multi-cloud with data sovereignty
- Supports a range of core trading technologies including Kdp+, a leader in high-performance time-series database systems from KX Systems
The Dell OneFS operating system does not rely on hardware as a critical part of the storage architecture. Instead, OneFS combines the three functions of traditional storage architectures—file system, volume manager, and data protection—into one unified software layer to create a single, intelligent file system that spans all nodes within a storage system. Unlike traditional storage systems that have a finite maximum size and must be replaced by a bigger storage array when the maximum performance or capacity is reached, a OneFS powered cluster can linearly expand, or scale out, performance and capacity to seamlessly increase the existing file system or volume into petabytes of capacity.
In addition to built-in availability, redundancy, security, data protection and replication with OneFS, the UDS portfolio provides protection from cyberattacks with integrated ransomware defense and smart AirGap. Superna Ransomware Defender detects suspicious behavioral patterns across multiple vectors, report on operations and event data in real-time, and block active threats.
The benefits for quantitative finance firms?
-
- Shortened model development time
- Faster analysis on larger, multi-day trading data
- Extreme performance for smaller real-time data
- Accelerated cycles of learning by bridging historical and real-time databases
- Ability to conform to regulatory standards and guarantee enterprise data protection
Discover how Dell Unstructured Data Solutions helps customers modernize quantitative analytics infrastructure to deliver high performance and extreme concurrency at scale. Contact your local Channel or UDS manager for additional information.
