---
title: "Upgrade to Aerospike Connect for Spark 6.0.0"
description: "Upgrade from Aerospike Connect for Spark 5.x to 6.0.0 on Apache Spark 4.1.x: platform requirements, connector configuration stability, and Spark 4 default changes."
---

# Upgrade to Aerospike Connect for Spark 6.0.0

> For the complete documentation index see: [llms.txt](https://aerospike.com/docs/llms.txt)
> 
> All documentation pages available in markdown.

This page describes compatibility and migration planning when you upgrade Aerospike Connect for Spark from 5.x (Spark 3.4.x or 3.5.x) to 6.0.0 on Apache Spark 4.1.x. Use it to identify connector changes, Spark platform risks, and cutover settings before you move production pipelines that read from or write to Aerospike.

## Prerequisites

Before you plan the upgrade, verify the following:

-   You target Apache Spark 4.1.x with Java 17 and Scala 2.13.
-   Your Aerospike Database cluster meets the [connector prerequisites](https://aerospike.com/docs/connectors/spark/installation#aerospike-connect-for-spark-prerequisites).
-   You have reviewed the [Aerospike Connect for Spark 6.0.0 release notes](https://aerospike.com/docs/connectors/spark/release-notes#connect-for-spark-600).

If you remain on Spark 3.4.x or 3.5.x, continue using Aerospike Connect for Spark 5.0.x. All `aerospike.*` configuration options from 5.0.x are preserved. No connector config renames or default changes are required. Platform requirements change to Java 17 (for Spark 4.1.x), Scala 2.13, and Spark 4.1.x. Read, write, `aerolookup`, filter pushdown, and TLS behavior are preserved.

Upgrade breakage risk comes from Spark 4 platform defaults, not connector API changes. The connector does not set or depend on Spark SQL runtime configs (`spark.sql.ansi.*`, timezone rebase, charset, or file handling). You are most exposed if you relied on Spark 3.5 permissive behavior without explicitly setting configs, and you:

-   Run SQL transforms after reading from Aerospike.
-   Write to Aerospike from DataFrames produced by permissive Spark 3.5 SQL.
-   Use intermediate Parquet, ORC, JDBC, or other file sources in the same pipeline.
-   Depend on silent nulls, wrap-around overflow, or legacy datetime rebasing.

## Connector configuration stability

| Area | Changed? | Notes |
| --- | --- | --- |
| `aerospike.*` keys | No | Same keys, aliases, and defaults |
| Spark configs set by connector | No | Connector reads only `aerospike.*` from `SparkConf` / `RuntimeConfig` |
| DataSource V2 API | Yes | Internal implementation change; not customer-facing |
| Java / Scala | Yes | Java 17 and Scala 2.13 required for Spark 4.1.x |
| `TypeConverter` | Stricter | Rejects numeric-to-string coercion; explicit overflow errors |

| Configuration | Default | Upgrade note |
| --- | --- | --- |
| `aerospike.flexible-schema` | `true` | Type mismatches on read become null (not exception) |
| `aerospike.write-batchsize` | `100` | Unchanged |
| `aerospike.client-poolsize` | `1` | Unchanged |
| `aerospike.partition-factor` | `8` | Unchanged |

## Spark configurations for initial cutover

To preserve Spark 3.5 behavior during initial cutover, set:

```properties
spark.sql.ansi.enabled=false

spark.sql.parquet.datetimeRebaseModeInRead=LEGACY

spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY

spark.sql.parquet.int96RebaseModeInRead=LEGACY

spark.sql.parquet.int96RebaseModeInWrite=LEGACY

spark.sql.maxSinglePartitionBytes=9223372036854775807
```

For Java 17 module access, set `spark.driver.extraJavaOptions` and `spark.executor.extraJavaOptions`. See [Java 17 module options for Spark 4.1.x](https://aerospike.com/docs/connectors/spark/installation#java-17-module-options-for-spark-41x).

After cutover, run pipelines with `spark.sql.ansi.enabled=true` in staging and fix failures with `try_cast`, `try_add`, and `try_divide`.

## Spark 4 default changes

### ANSI SQL compliance (highest risk)

`spark.sql.ansi.enabled` defaults to `true` in Spark 4 (was `false` in Spark 3.5). Invalid casts, arithmetic overflow, division by zero, and out-of-range array access return exceptions instead of null. Connector read conversion is ANSI-independent, but downstream SQL on Aerospike data is not.

### Timezone and datetime rebase (medium–high risk)

Legacy `spark.sql.legacy.parquet.*` and `spark.sql.legacy.avro.*` rebase keys are removed. Migrate to non-legacy names (`spark.sql.parquet.*`, `spark.sql.avro.*`). Direct Aerospike read and write is unaffected because the connector stores dates and timestamps as epoch milliseconds.

| Removed (Spark 3.5) | Replacement (Spark 4) |
| --- | --- |
| `spark.sql.legacy.parquet.int96RebaseModeInWrite` | `spark.sql.parquet.int96RebaseModeInWrite` |
| `spark.sql.legacy.parquet.datetimeRebaseModeInWrite` | `spark.sql.parquet.datetimeRebaseModeInWrite` |
| `spark.sql.legacy.parquet.int96RebaseModeInRead` | `spark.sql.parquet.int96RebaseModeInRead` |
| `spark.sql.legacy.avro.datetimeRebaseModeInWrite` | `spark.sql.avro.datetimeRebaseModeInWrite` |
| `spark.sql.legacy.avro.datetimeRebaseModeInRead` | `spark.sql.avro.datetimeRebaseModeInRead` |

### Character encoding (conditional risk)

Spark 4 restricts `encode()` and `decode()` charset support. The connector uses UTF-8 for string bins and is unaffected unless your pipelines call `encode()` or `decode()` on Aerospike string columns. To restore legacy behavior, set `spark.sql.legacy.javaCharsets=true` and `spark.sql.legacy.codingErrorAction=true`.

### File-handling behavior (low–medium for Aerospike-only pipelines)

No direct impact on `spark.read.format("aerospike")`. In multi-source ETL pipelines, `spark.sql.maxSinglePartitionBytes` now defaults to `128m` (was `Long.MaxValue`), corrupt file handling uses `spark.sql.files.ignoreCorruptFiles` (not `spark.files.*`), and map key `-0.0` is normalized unless `spark.sql.legacy.disableMapKeyNormalization=true`.

## Next steps

-   [Install Aerospike Connect for Spark](https://aerospike.com/docs/connectors/spark/installation) — download 6.0.0 artifacts and set Java 17 JVM options.
-   [Configure Aerospike Connect for Spark](https://aerospike.com/docs/connectors/spark/configuration) — review `aerospike.*` settings for your pipelines.
-   [Troubleshoot Aerospike Connect for Spark](https://aerospike.com/docs/connectors/spark/troubleshooting) — resolve Java version and `InaccessibleObjectException` errors.