A real-time data store to support Snowflake and advanced analytics on AWS

This global financial services organization helps millions of individuals and thousands of organizations and wealth management firms achieve their financial goals by delivering innovative investment solutions digitally. Their focus on innovation provides insight across the markets with the stability for long term success for their clients.

Ingest real-time interactions and market data to support AI/ML decisions

The firm’s business units generated and contributed to an enterprise data lake built on top of AWS and Snowflake. This solution was used to create AI/ML models to guide personalization, market risk analysis, and other advanced analytics at scale. The initial solution was sufficient for relational, low volume, internal-only applications. However, the firm had a need to provide more real-time and thus accurate data for decision support with these analytics as well as scalable APIs to serve the data across their business units in order to service over half a million customers at scale. As a data lake and data warehouse, Snowflake was unable to process in real-time at scale. The firm realized it needed to find a high performance, low latency solution that could ingest and serve up data at scale for real-time decisions.

Specifically, the financial services firm needed a data platform that could handle different types of data, including:

• Near real-time data from customer calls or online interactions, such as web click data, chat data, interaction data

• Real-time data such as trade data or other transactional data like operational, and back-office data 

• Batch and micro-batch data such as market data that comes into the system when the market closes at the end of the day

• High performance ingestion

  Handle real-time and micro batch data from customer interactions and transactions, then used to train, validate, and score AI/ML models

• Low latency data store

  Scalable APIs to serve data to applications used by customers, clients, and internal users

• Integration with Snowflake and AWS

  Work seamlessly with Snowflake’s enterprise data analytics platform and AWS

• Enterprise cloud computing and security requirements

  Must be easy to set up, manage, and comply with requirements, including the ability to rehydrate nodes every 60 days with zero downtime

Near real-time ingestion and low latency data store on Snowflake and AWS

The financial services firm realized that trying to put all their working data in-memory would be incredibly expensive.  They needed a solution that would give them flexibility, high performance, but would also meet strict financial security and enterprise requirements. Able to support multiple cloud service providers as well as integrate with Snowflake and AWS, Aerospike provided the firm with what they needed in a cloud data platform supporting their AI/ML models and advanced analytics, including delivering:

• Phenomenal performance

  Easily handling any workload the firm needed to run, whether it was a read-heavy workload, write-heavy workload, or mixed workload.

• Multiple storage options

  Has the flexibility to store data in memory, on flash, on drive, file systems, or more.

• Multi-model

  While currently being used as a key-value store, the firm plans on expanding to include document and graph store capabilities.

• Always-on

  Able to comply with enterprise cloud computing and security requirements with an automated rehydration process that replaces entire nodes with zero downtime.