Aerospike Will No Longer Support the Large Data Type (LDT) Feature

In 2013, Aerospike introduced a new feature known as Large Data Types (LDTs) to enable us to handle data that exceeded our row size limit. Effective immediately, we will no longer actively support the LDT feature and will eventually remove the API. The exact deprecation and removal timeline will depend on customer and community requirements. Instead of LDTs, we advise that you use our newer List and SortedMap APIs, which are now available in all Aerospike-supported clients at the General Availability level.

The LDT BackstoryTo maximize predictability and performance, Aerospike has a database row size limit of 1MB. Before we introduced LDTs, some larger and more complex data structures with outlying data could exceed this limit, which would effectively break the row. The LDT system was designed to take advantage of 4KB storage blocks and maintain types in an on-disk format that is efficient for both large and small objects. The LDT feature addressed the row size limit problem but unfortunately introduced other unforeseen problems.

LDT Introduces New Problems

Our first LDT-related problem was the API, which was built on Aerospike’s User-Defined Function (UDF) language. The API did not integrate with our existing APIs as seamlessly as we had hoped, and we quickly realized that we needed to provide a more natural API. We were able to—and had to—integrate cleanly with each language-specific data type, in that native language.

Our second LDT-related problem was the implementation, which was also built on Aerospike’s UDF language. Code complexity was higher than expected due to atomicity and correctness concerns, and LDT performance never quite met Aerospike standards. Accordingly, some correctness bugs have proven very difficult to resolve.

Making Progress with New List and SortedMap Functionality

Aerospike’s new List and SortedMap features solve our first problem by introducing a better API. We have carefully designed and vetted this API to be easy to use and provide all required functionality. The new APIs also greatly improve implementation performance as the replacement system is written in C and integrates with the standard Aerospike internal type system.

However, the new implementation does not solve the problem of the 1MB Aerospike database row size limit. A future key feature of the product will be an enhanced implementation that transcends the 1MB limit for a number of types. We will keep the community apprised of our progress.

LDTs Should No Longer Be Used

Currently, we advise that all customers avoid using the LDT interfaces and instead transition the code that uses these interfaces to the List and SortedMap APIs. We have found that most applications can use a variety of techniques such as bucketing to provide higher performance and reliability using the new APIs. Please check out our documentation for detailed information on LDT alternatives.

If you have any questions or need further assistance making these changes, please reach out for support.