A real-time data store to support Snowflake and advanced analytics on AWS
About the company
This global financial services organization helps millions of individuals and thousands of organizations and wealth management firms achieve their financial goals by delivering innovative investment solutions digitally. Their focus on innovation provides insight across the markets with the stability for long term success for their clients.
Ingest real-time interactions and market data to support AI/ML decisions
The firm’s business units generated and contributed to an enterprise data lake built on top of AWS and Snowflake. This solution was used to create AI/ML models to guide personalization, market risk analysis, and other advanced analytics at scale. The initial solution was sufficient for relational, low volume, internal-only applications. However, the firm had a need to provide more real-time and thus accurate data for decision support with these analytics as well as scalable APIs to serve the data across their business units in order to service over half a million customers at scale. As a data lake and data warehouse, Snowflake was unable to process in real-time at scale. The firm realized it needed to find a high performance, low latency solution that could ingest and serve up data at scale for real-time decisions.
Specifically, the financial services firm needed a data platform that could handle different types of data, including:
Near real-time data from customer calls or online interactions, such as web click data, chat data, interaction data
Real-time data such as trade data or other transactional data like operational, and back-office data
Batch and micro-batch data such as market data that comes into the system when the market closes at the end of the day
High performance ingestion
Handle real-time and micro batch data from customer interactions and transactions, then used to train, validate, and score AI/ML models
Low latency data store
Scalable APIs to serve data to applications used by customers, clients, and internal users
Integration with Snowflake and AWS
Work seamlessly with Snowflake’s enterprise data analytics platform and AWS
Enterprise cloud computing and security requirements
Must be easy to set up, manage, and comply with requirements, including the ability to rehydrate nodes every 60 days with zero downtime
Near real-time ingestion and low latency data store on Snowflake and AWS
The financial services firm realized that trying to put all their working data in-memory would be incredibly expensive. They needed a solution that would give them flexibility, high performance, but would also meet strict financial security and enterprise requirements. Able to support multiple cloud service providers as well as integrate with Snowflake and AWS, Aerospike provided the firm with what they needed in a cloud data platform supporting their AI/ML models and advanced analytics, including delivering:
Phenomenal performance
Easily handling any workload the firm needed to run, whether it was a read-heavy workload, write-heavy workload, or mixed workload.
Multiple storage options
Has the flexibility to store data in memory, on flash, on drive, file systems, or more.
Multi-model
While currently being used as a key-value store, the firm plans on expanding to include document and graph store capabilities.
Always-on
Able to comply with enterprise cloud computing and security requirements with an automated rehydration process that replaces entire nodes with zero downtime.
Advanced analytics, scalable APIs, and real-time performance with Aerospike
The firm takes full advantage of Aerospike’s flexibility, performance capabilities, and AWS integration to run their near real-time ingestion layer on top of Snowflake and serve up data via APIs to their customers, clients, and internal users.
They run Aerospike nodes in three AWS availability zones per region, with multi-region active-active replication via Aerospike’s cross datacenter replication (XDR) capabilities. With the flexibility to keep the primary index in-memory while putting the data in 7.5 TB, EC2 solid state drives, Aerospike is able to deliver impressive performance and scalability results.