Optimizing fraud detection: Lessons from Caulis’ database migration
Learn how Caulis tackled scaling challenges in fraud detection by migrating their database from Redis to Aerospike and uncover the key learnings and factors that led to a successful transition.
As the saying goes, computer security professionals have to be perfect all the time, while fraudsters only have to be correct once. Consequently, fraud detection systems must remain robust and agile. Caulis, a Japanese fraud detection company, faced challenges in scaling its fraud detection solution. Its path from using Redis to adopting Aerospike as its primary database offers insights into scaling such systems. This blog post goes into the lessons learned from Caulis's database migration and the factors that contributed to its success.
Introduction to Caulis and FraudAlert
Caulis’ main product is FraudAlert, a cloud-based fraud detection solution that uses artificial intelligence to analyze user behavior and detect unusual and potentially fraudulent access patterns. It authenticates customer touch interactions across industries and devices, notifying clients of high-risk activities in real time. Additionally, Caulis collaborates on various fronts to enhance fraud prevention, with plans to expand its capabilities to ATMs and IoT devices, explained Caulis Product Manager Yasuyuki Hayashi at the recent Real-time Data Summit.
FraudAlert starts with straightforward rule-based detection and progressively incorporates machine learning to enhance its capabilities as more data becomes available. This allows the system to adapt and identify complex and emerging fraud patterns, reducing false positives and operational costs. FraudAlert offers seven services: login detection, phishing site detection, integration with information provided by select electric power companies, transfer detection, screen transition detection, new account opening detection, and consulting on how to use the product for monitoring operations.
Moreover, FraudAlert has to perform in real time. “Criminals act quickly, making timely notifications essential to mitigate damage,” Hayashi says.
Redis challenges
Initially, Caulis chose Redis due to its speed and ease of schema management. However, as the company’s data volume grew, several issues emerged.
Scalability: Horizontal and vertical scaling were problematic. "Scaling out and scaling in could not be achieved without downtime, making continuous service challenging,” Hayashi says.
Data integrity: Backup files sometimes crashed. "We faced long recovery times and data errors due to backup file crashes," Hayashi explains.
Complexity: Clustering was complex, and re-adding instances often led to instability. "Understanding and correctly using the clustering system was complex despite seeking advice from Redis professionals and conducting various tests over two years," says Hayashi.
“Redis is outstanding for caching and handling non-permanent data, but our use case involved semi-permanent data, making Redis unsuitable,” he added.
These challenges prompted Caulis to look for a better solution – and the future of the company was at stake. “Our product needed work to fit our target market, making cost a significant factor,” Hayashi says. “A wrong decision could have led to bankruptcy.”
Key requirements for the new database
Caulis identified several critical capabilities for its new database:
Scalability: Both horizontal and vertical scaling without downtime
Reliable backups: Easy-to-manage data integrity and quick recovery
Cost performance: Cost-efficient management for operations, migrations, and downtime
Evaluation of potential databases
Caulis evaluated three potential databases to replace Redis: Aerospike, Amazon DynamoDB, and Amazon Aurora. Each option had advantages and disadvantages.
Aerospike: Presented potential operation and maintenance challenges but offered robust support as well as easy and transparent cost calculation based on data size.
DynamoDB: Offered reliable and fast performance with AWS support but posed the risk of runaway costs due to usage-based pricing.
Aurora: Provided easy cost predictability and enterprise support but, as a relational database, required rigid schema management.
The decision to migrate to Aerospike
Caulis’ evaluation put maintenance, operation, and support evenly between Aerospike, DynamoDB, and Aurora. They also wanted to ensure they weren’t sacrificing speed by leaving the in-memory Redis architecture. Using AWS’ Traffic Mirroring testing approach, they verified that Aerospike was indeed able to match Redis’ speed.
While initially unfamiliar with Aerospike, Caulis also gained confidence in its ability to be deployed rapidly as well as Aerospike’s ability to provide rapid support.
Ultimately, the biggest difference in the evaluation was the total cost of ownership. Caulis took a thorough view of this, examining initial costs, operating costs, migration costs, and the potential costs of downtime.
“Cost performance was a deciding factor, including development costs,” Hayashi says. “Aerospike provided satisfactory speed, stability, ease of operation, and reliability. I was relieved to be free from our data worries.”
Benefits realized with Aerospike
Migrating to Aerospike brought several benefits.
Enhanced performance: Achieved real-time processing with 1700 to 2000 transactions (queries) per second (TPS). "The speed is extremely fast, averaging 100 milliseconds with a 99.9% latency of 500 milliseconds,” Hayashi reports.
Scalability and stability: Aerospike provided the horizontal and vertical scaling Caulis needed, without downtime.
Cost efficiency: Provided lower operational and development costs compared with other solutions.
Lessons learned
Caulis’s journey provides several takeaways.
Thorough evaluation: Assess potential databases against your specific use case and requirements. "It's essential to choose a database based on your use case, business needs, and organizational size," Hayashi advises.
Comprehensive testing: Use real-world traffic to test and validate the new system’s performance and reliability. "We verified the fast writes and ensured consistency through extensive testing," says Hayashi.
Strategic planning: Plan for potential challenges in migration and ensure robust support and maintenance structures are in place.
Aerospike: The right choice for real-time fraud detection
Optimizing fraud detection systems requires a real-time database that can scale with business needs. Caulis’s successful migration from Redis to Aerospike highlights the importance of scalability, reliability, and cost efficiency. By sharing its experiences, Caulis provides helpful advice for other organizations facing similar challenges in enhancing their capabilities around fraud detection and other low-latency, high-throughput use cases. "Databases including Aerospike, Redis, and relational database management systems (such as Aurora) have their strengths,” Hayashi says. "It's crucial to choose a database based on your specific needs."
