Skip to content

Upgrade to Aerospike Connect for Spark 6.0.0

For the complete documentation index see: llms.txt

All documentation pages available in markdown.

This page describes compatibility and migration planning when you upgrade Aerospike Connect for Spark from 5.x (Spark 3.4.x or 3.5.x) to 6.0.0 on Apache Spark 4.1.x. Use it to identify connector changes, Spark platform risks, and cutover settings before you move production pipelines that read from or write to Aerospike.

Prerequisites

Before you plan the upgrade, verify the following:

If you remain on Spark 3.4.x or 3.5.x, continue using Aerospike Connect for Spark 5.0.x. All aerospike.* configuration options from 5.0.x are preserved. No connector config renames or default changes are required. Platform requirements change to Java 17 (for Spark 4.1.x), Scala 2.13, and Spark 4.1.x. Read, write, aerolookup, filter pushdown, and TLS behavior are preserved.

Upgrade breakage risk comes from Spark 4 platform defaults, not connector API changes. The connector does not set or depend on Spark SQL runtime configs (spark.sql.ansi.*, timezone rebase, charset, or file handling). You are most exposed if you relied on Spark 3.5 permissive behavior without explicitly setting configs, and you:

  • Run SQL transforms after reading from Aerospike.
  • Write to Aerospike from DataFrames produced by permissive Spark 3.5 SQL.
  • Use intermediate Parquet, ORC, JDBC, or other file sources in the same pipeline.
  • Depend on silent nulls, wrap-around overflow, or legacy datetime rebasing.

Connector configuration stability

AreaChanged?Notes
aerospike.* keysNoSame keys, aliases, and defaults
Spark configs set by connectorNoConnector reads only aerospike.* from SparkConf / RuntimeConfig
DataSource V2 APIYesInternal implementation change; not customer-facing
Java / ScalaYesJava 17 and Scala 2.13 required for Spark 4.1.x
TypeConverterStricterRejects numeric-to-string coercion; explicit overflow errors
ConfigurationDefaultUpgrade note
aerospike.flexible-schematrueType mismatches on read become null (not exception)
aerospike.write-batchsize100Unchanged
aerospike.client-poolsize1Unchanged
aerospike.partition-factor8Unchanged

Spark configurations for initial cutover

To preserve Spark 3.5 behavior during initial cutover, set:

spark.sql.ansi.enabled=false
spark.sql.parquet.datetimeRebaseModeInRead=LEGACY
spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY
spark.sql.parquet.int96RebaseModeInRead=LEGACY
spark.sql.parquet.int96RebaseModeInWrite=LEGACY
spark.sql.maxSinglePartitionBytes=9223372036854775807

For Java 17 module access, set spark.driver.extraJavaOptions and spark.executor.extraJavaOptions. See Java 17 module options for Spark 4.1.x.

After cutover, run pipelines with spark.sql.ansi.enabled=true in staging and fix failures with try_cast, try_add, and try_divide.

Spark 4 default changes

ANSI SQL compliance (highest risk)

spark.sql.ansi.enabled defaults to true in Spark 4 (was false in Spark 3.5). Invalid casts, arithmetic overflow, division by zero, and out-of-range array access return exceptions instead of null. Connector read conversion is ANSI-independent, but downstream SQL on Aerospike data is not.

Timezone and datetime rebase (medium–high risk)

Legacy spark.sql.legacy.parquet.* and spark.sql.legacy.avro.* rebase keys are removed. Migrate to non-legacy names (spark.sql.parquet.*, spark.sql.avro.*). Direct Aerospike read and write is unaffected because the connector stores dates and timestamps as epoch milliseconds.

Removed (Spark 3.5)Replacement (Spark 4)
spark.sql.legacy.parquet.int96RebaseModeInWritespark.sql.parquet.int96RebaseModeInWrite
spark.sql.legacy.parquet.datetimeRebaseModeInWritespark.sql.parquet.datetimeRebaseModeInWrite
spark.sql.legacy.parquet.int96RebaseModeInReadspark.sql.parquet.int96RebaseModeInRead
spark.sql.legacy.avro.datetimeRebaseModeInWritespark.sql.avro.datetimeRebaseModeInWrite
spark.sql.legacy.avro.datetimeRebaseModeInReadspark.sql.avro.datetimeRebaseModeInRead

Character encoding (conditional risk)

Spark 4 restricts encode() and decode() charset support. The connector uses UTF-8 for string bins and is unaffected unless your pipelines call encode() or decode() on Aerospike string columns. To restore legacy behavior, set spark.sql.legacy.javaCharsets=true and spark.sql.legacy.codingErrorAction=true.

File-handling behavior (low–medium for Aerospike-only pipelines)

No direct impact on spark.read.format("aerospike"). In multi-source ETL pipelines, spark.sql.maxSinglePartitionBytes now defaults to 128m (was Long.MaxValue), corrupt file handling uses spark.sql.files.ignoreCorruptFiles (not spark.files.*), and map key -0.0 is normalized unless spark.sql.legacy.disableMapKeyNormalization=true.

Next steps