---
title: "Configure Aerospike Connect for Spark"
description: "Configure Aerospike Connect for Spark: Read/write parameters, TTL, TLS, schema inference, and performance tuning."
---

# Configure Aerospike Connect for Spark

> For the complete documentation index see: [llms.txt](https://aerospike.com/docs/llms.txt)
> 
> All documentation pages available in markdown.

Set the configuration parameters for reads and writes with Spark’s `option()` function. For example, if you are using the Spark connector version 2.7.0 or later, here’s how you specify the Aerospike set that you want to read from:

```plaintext
as_data=spark \

.read \

.format("aerospike") \

.option("aerospike.read-set", "natality").load()
```

If you are using the Spark connector version 2.6.0 or earlier, set the format to `com.aerospike.spark.sql`, like this:

```plaintext
as_data=spark \

.read \

.format("com.aerospike.spark.sql") \

.option("aerospike.read-set", "natality").load()
```

You can use `spark.conf()` to set other properties, as in these examples:

```plaintext
spark.conf.set("aerospike.namespace", AS_NAMESPACE)

spark.conf.set("aerospike.seed-nodes", AS_CONNECTION_STRING)

spark.conf.set("aerospike.keyPath", AS_FEATURE_KEY_PATH)
```

## Feature-key

A feature-key file with the feature key `spark connector` or the feature key `raf-realtime-analysis-framework` turned on is required for using the Spark connector. If you try to use the Spark connector without a feature key, the connector logs an error message in Spark’s log file and does not run.

-   For Spark connector versions 2.8 (for use with Spark 2.4._x_) and 3.1 (for use with Spark 3._x_), you can include the feature key in the feature-key file for your Aerospike cluster. If that is not feasible, then use either the `aerospike.keyPath` or `aerospike.featureKey` properties in your Spark applications.
-   For earlier versions of the Spark connector, you must use either the `aerospike.keyPath` or `aerospike.featureKey` properties in your Spark applications.

| Spark Connector Version | How to give the Spark connector access to a feature key? |
| --- | --- |
| 3.1 and later | Include the feature-key file features.conf in the Aerospike cluster. |
| 2.8 | Include the feature-key file features.conf in the Aerospike cluster. |
| Earlier than 3.1 and 2.8 | Specify the feature key in the Spark application using `aerospike.keyPath` or `aerospike.featureKey` property. |

`aerospike.featureKey`

**Description:** Allows you to set the content of a feature-key file as a string in a Spark application.

::: note
This flag was discontinued in version 4.0.0. Use `aerospike.keyPath` or include the feature-key file in the Aerospike cluster for connector versions 3.1+ and 2.8+.
:::

Here is an example of this configuration property being used with connector versions prior to 4.0.0:

```plaintext
val df=spark

.read

.format("aerospike").

.option("aerospike.seedhost", "cluster1:3000")

.option("aerospike.set", "cluster1")

.option("aerospike.featureKey", "my-feature-file-content")

.load()
```

Your code can read the value in many ways, such as from an environment variable.

**Possible values:** N/A

**Default value:** N/A

**Required?:** Yes (for connector versions prior to 4.0.0), if you are not using `aerospike.keyPath` and the feature-key file is not located in the path that is the default value for `aerospike.keyPath`. The connector must have access to either the feature key `spark connector` or the feature key `raf-realtime-analysis-framework`.

---

`aerospike.keyPath`

**Description:** Path to an Aerospike feature-key file. A copy of this file must be present on all nodes of your Spark cluster, including the master node, at this location.

::: note
This flag was discontinued in version 4.0.0. For connector versions 3.1+ and 2.8+, include the feature-key file in the Aerospike cluster instead.

If you plan to store the feature-key file in HDFS, set the property `aerospike.hdfsCoreSiteXmlPath` to the path of the `core-site.xml` file.
:::

**Possible values:** N/A

**Default value:** `/etc/aerospike/features.conf` The connector uses this default value when neither `aerospike.keyPath` nor `aerospike.featureKey` is specified.

**Required?:** Yes, if you are not using `aerospike.featureKey` and the feature-key file is not at the default path.

---

`aerospike.hdfsCoreSiteXmlPath`

::: note
\*This flag was discontinued in version 4.0.0.
:::

**Description:** Absolute path to an HDFS `core-site.xml` file.

**Possible values:** N/A

**Default value:** N/A

**Required?:** Yes, if you are using `aerospike.keyPath` and are storing the feature-key file in HDFS.

---

## Initial configuration and tuning

`aerospike.infered-digest-type`

**Alias** `aerospike.digestType` is deprecated for versions 5.0.0 and later.

**Description:** The data type of digests. You use this property together with the `aerospike.schema-scan-record` property.

**Possible values:** `string`, `binary`

**Default value:** `binary`

**Required?:** No

---

`aerospike.schema-scan-record`

**Alias** `aerospike.schema.scan` is deprecated for versions 5.0.0 and later.

**Description:** The number of records for the connector to scan to infer the schema of the data in a DataFrame. The connector infers the schema for every query. Aerospike recommends using schema inference when you are using complex types, such as maps and lists.

-   If the records in your Aerospike set use the same schema, you can set a lower value.
-   If records use more than one schema, you can set a higher value.

The more variety there is in the schemas, the higher you should set the value. If the value is too low in this case, the number of records that the connector scans cannot adequately account for variations in the schema. If the value is too high, the master node of the Spark cluster can run out of memory because schema inference uses resources on that node.

**Possible values:** N/A

**Default value:** 100

**Required?:** Yes, if you do not provide the schema to use.

---

`aerospike.seed-nodes`

**Alias** `aerospike.seedhost` is deprecated for versions 5.0.0 and later.

**Description:** The nodes from which the client application should obtain a list of the locations and ports of all of the nodes in the Aerospike cluster.

**Possible values:** Comma-separated host:port pairs, like these: `host1:port1,host2:port2`

You can also operate between multiple clusters by loading from one Aerospike cluster and writing the DataFrame into another.

**Default value:** `localhost:3000`

**Required?:** Yes, if the Aerospike cluster is not running locally.

---

`aerospike.read-set`

**Alias** `aerospike.set` is deprecated for versions 5.0.0 and later.

**Description:** The Aerospike set from which records are to be read into a Spark DataFrame.

**Possible values:** N/A

**Default value:** N/A

**Required?:** Yes, for read operations

---

`aerospike.clientpolicy.useservicesalternate`

**Alias** `aerospike.useServicesAlternate` is deprecated for versions 5.0.0 and later.

**Description:** This option specifies how the info request is made during cluster tending. See the [Java client documentation](https://javadoc.io/static/com.aerospike/aerospike-client/6.0.1/com/aerospike/client/policy/ClientPolicy.html#useServicesAlternate) for a detailed description. This flag is available in Spark connector 2.9.0 in the v2.x series and in Spark connector >= 3.5.0 in v3.x series.

**Possible values:** `true`, `false`

**Default value:** `false`

**Required?:** No

---

`aerospike.write-with-digest`

**Alias** `aerospike.updateByDigest` is deprecated for versions 5.0.0 and later.

**Description:** This option specifies that updates are done by digest with the value in the column specified in `option("aerospike.write-with-digest", "Digest")`

**Possible values:** N/A

**Default value:** N/A

**Required?:** Yes, if no value is given for `aerospike.write-with-key`.

---

`aerospike.write-with-key`

**Alias** `aerospike.updateByKey` is deprecated for versions 5.0.0 and later.

**Description:** This option specifies that updates are done by primary key with the value in the column specified in `option("aerospike. updateByKey","[column-name]")` This configuration parameter supports nested keys. Such key types must be accepted by a client application. Consider this simple Structfield: `{A:{B:1}}`. A client app can use `option("aerospike.write-with-key", "A.B")` to update records with the value `1`. Use `.` to separate paths in a structure. Ensure that names in a Structfield do not include periods.

**Possible values:** N/A

**Default value:** N/A

**Required?:** Yes, if no value is given for `aerospike.write-with-digest`

---

`aerospike.write-batchsize`

**Alias** `aerospike.write.batchsize` is deprecated for versions 5.0.0 and later.

**Description:** Connector supports both [batch](https://aerospike.com/docs/develop/learn/batch) and [non-batched](https://javadoc.io/static/com.aerospike/aerospike-client/6.1.6/com/aerospike/client/AerospikeClient.html#put-com.aerospike.client.policy.WritePolicy-com.aerospike.client.Key-com.aerospike.client.Bin...-) records writes. Batch writes requires Database 6.0 and later releases. When the internal write buffer reaches the specified value, the connector blocks subsequent requests until all previously submitted write requests are resolved. `aerospike.write-batchsize` affects the write throughput (writes per second) and the latency of the write workload. It applies to each Spark partition individually. `aerospike.write-batchsize` begins operating when Aerospike Database 6.0 or later is in use.

If you set the value to 20,000 records, the connector checks the status of each write operation after writing each batch of 20,000 records to your Aerospike cluster. A large value ensures that write throughput (writes per second)) is high. However, the larger the value, the longer the delay before the connector checks for and handles errors. If the value is set to 1, then error checking and handling occurs much more frequently, and therefore the latency of the write workload is much higher.

**Possible values:** N/A

**Default value:** 100

**Required?:** No

---

`aerospike.max-batch-inflight`

**Description:** Controls the maximum number of batch write requests that can be in flight (submitted but not yet completed) at any given time per Spark partition. When this limit is reached, the connector blocks until one or more in-flight batches complete before submitting new ones. This flag was introduced in connector version 5.0.0.

**Possible values:** N/A

**Default value:** 10

**Required?:** No

---

`aerospike.booleanbin`

**Description:** Specifies whether the records that the connector reads from your Aerospike cluster use the Boolean data type for values in Boolean bins, rather than using integer values of 0 and 1. Ensure that your Aerospike cluster is Aerospike Database Enterprise Edition version 5.6.0.3 or later before you set the value of this configuration parameter to `true`.

Setting this value to true can affect client applications that query the data after it is loaded into Spark. Those applications might expect integer values from a Boolean column, rather than Boolean values. Also, if the application that uses the connector to load data into Spark currently filters on a Boolean column and the filter is defined based on the integer values 0 and 1, and the data in the column is a mix of integer values and Boolean values, the filter needs to be revised. If filtering takes place on the Aerospike cluster, the application should follow this logic:

-   Query the Aerospike cluster by filtering on integer values in the Boolean column.
-   Query the Aerospike cluster by filtering on Boolean values in the Boolean column.
-   Create a union of the result sets in the DataFrame in Spark.

**Possible values:** `true`, `false`

**Default value:** `false`

**Required?:** No

---

`aerospike.writepolicy.commitlevel`

**Alias** `aerospike.commitLevel` is deprecated for versions 5.0.0 and later.

**Description:** The consistency level for Aerospike to apply when committing transactions from a client application.

**Possible values:** `CommitLevel.COMMIT_ALL` The database waits to commit a transaction until after committing on a master mode and all of its replicas. `CommitLevel.COMMIT_MASTER` The database waits to commit a transaction until after committing on a master node only, and doesn’t wait until after committing on the node’s replicas, too.

**Default value:** `CommitLevel.COMMIT_ALL`

**Required?:** No

---

`aerospike.digest-column`

**Alias** `aerospike.digestColumn` is deprecated for versions 5.0.0 and later.

**Description:** Name of the column in the DataFrame that contains digests for corresponding Aerospike records.

**Possible values:** N/A

**Default value:** `__digest`

**Required?:** No

---

`aerospike.expiry-column`

**Alias** `aerospike.expiryColumn` is deprecated for versions 5.0.0 and later.

**Description:** A read-only configuration that defines the DataFrame column holding expiration dates for Aerospike records. These dates are calculated as the sum of the record’s write time and TTL, expressed in seconds since the Citrusleaf epoch (00:00:00 UTC on 1 Jan 2010).

**Possible values:** N/A

**Default value:** `__expiry`

**Required?:** No

---

`aerospike.generation-column`

**Alias** `aerospike.generationColumn` is deprecated for versions 5.0.0 and later.

**Description:** Name of the column in the DataFrame for the record-generation counter in corresponding Aerospike records.

**Possible values:** N/A

**Default value:** `__generation`

**Required?:** No

---

`aerospike.writepolicy.generationpolicy`

**Alias** `aerospike.generationPolicy` is deprecated for versions 5.0.0 and later.

**Description:** How to handle record writes based on the record-generation counter. Generation is the number of times an Aerospike record has been modified. When a record is created, its counter is set to 1. For more information, see the “Write Generation Policy” section in [Policies](https://aerospike.com/docs/database/learn/policies).

**Possible values:** `EXPECT_GEN_EQUAL` Update/delete record if expected generation is equal to server generation. `EXPECT_GEN_GT` Update/delete record if expected generation greater than the server generation. `NONE` Do not use record generation to restrict writes.

**Default value:** `NONE`

**Required?:** No

---

`aerospike.key-column`

**Alias** `aerospike.keyColumn` is deprecated for versions 5.0.0 and later.

**Description:** Name of the column in the DataFrame that contains keys for corresponding Aerospike records.

**Possible values:** N/A

**Default value:** `__key`

**Required?:** No

---

`aerospike.infered-key-type`

**Alias** `aerospike.keyType` is deprecated for versions 5.0.0 and later.

**Description:** Data type of the primary key in a DataFrame. You can use this parameter if the connector is inferring the schema of the data in the DataFrame.

**Possible values:** `int`, `short`, `long`, `double`, `float`, `date`, `timestamp`, `string`, `binary`

**Default value:** `string`

**Required?:** No

---

`aerospike.log-level`

**Alias** `aerospike.log.level` is deprecated for versions 5.0.0 and later.

**Description:** Aerospike connector logging level.

**Possible values:** `info`, `debug`, `error`, `all`, `warn`, `trace`, `off`, `fatal`

**Default value:** `warn`

**Required?:** No

---

`aerospike.namespace`

**Description:** Name of the Aerospike namespace to read from and write to.

**Possible values:** N/A

**Default value:** `test`

**Required?:** No

---

`aerospike.flexible-schema`

**Alias** `aerospike.schema.flexible` is deprecated for versions 5.0.0 and later.

**Description:** Spark SQL assumes that the records in a Spark table (that maps to an Aerospike set) use a single schema. However, as a NoSQL database, Aerospike does not require records in a set to use a single schema. An Aerospike bin (which maps to a column in a Spark DataFrame) within an Aerospike set (which maps to a Spark table) could contain values that are of multiple data types. Schemas in Spark can be inferred or user-provided. The configuration parameter `aerospike.flexible-schema`, when set to `true`, reconciles this incompatibility. For more information, see “Flexible schemas” below.

**Possible values:** `true`, `false`

**Default value:** `true`

**Required?:** No

---

`aerospike.clientpolicy.sendkey`

**Alias** `aerospike.sendKey` is deprecated for versions 5.0.0 and later.

**Description:** Specifies whether the connector should include the primary key when writing a record to an Aerospike database. If this value is `true`, the value of the primary key is stored in the bin named ‘\_\_key’ in the database.

**Possible values:** N/A

**Default value:** `false`

**Required?:** No

---

`aerospike.clientpolicy.sockettimeout`

**Alias** `aerospike.sockettimeout` is deprecated for versions 5.0.0 and later.

**Description:** Server-side socket timeout (in milliseconds) for query/scan operations. (0 = no timeout)

**Possible values:** N/A

**Default value:** 30000

**Required?:** No

---

`aerospike.clientpolicy.timeout`

**Alias** `aerospike.timeout` is deprecated for versions 5.0.0 and later.

**Description:** Timeout (in milliseconds) for database commands invoked by the connector.

**Possible values:** N/A

**Default value:** 1000

**Required?:** No

---

`aerospike.ttl-column`

**Alias** `aerospike.ttlColumn` is deprecated for versions 5.0.0 and later.

**Description:** Name of the column in the DataFrame for the TTL value in corresponding Aerospike records, which sets the write policy expiration for records.

**Possible values:** N/A

**Default value:** `__ttl`

**Required?:** No

---

`aerospike.partial-update`

**Alias** `aerospike.update.partial` is deprecated for versions 5.0.0 and later.

**Description:** Specifies whether to allow partial updates to existing Aerospike records. A partial update affects only a subset of the bins in a record, with the rest of the bins in the update set to null. A null value indicates that the bin is not being updated and that the current value in the record should remain as it is. By design, however, if a record update includes null bins, Aerospike deletes the corresponding bins from the record. The `aerospike.partial-update` configuration parameter, when set to `true`, prevents the corresponding bins from being deleted. For example, suppose a record contains the bins `prod_ID`, which contains the primary key, `prod_name`, and `prod_price`. A partial update for a particular `prod_ID` sets `prod_name` to `null` and updates the value of `prod_price`. If `aerospike.partial-update` is set to the default, `false`, Aerospike deletes the bin `prod_name` from the record and updates the value in the bin `prod_price`. If `aerospike.partial-update` is set to `true`, then Aerospike leaves the bin `prod_name` as it is and updates the value in the bin `prod_price`. You can set the value to `true` only when the value of `aerospike.writepolicy.recordexistsaction` is set to `UPDATE` or `UPDATE_ONLY`. This feature allows for the deletion of records in the Aerospike database only if all bins are null and when using the `UPDATE` or `UPDATE_ONLY` policy.

**Possible values:** `true`, `false`

**Default value:** `false`

---

`aerospike.writepolicy.recordexistsaction`

**Alias** `aerospike.write.mode` is deprecated for versions 5.0.0 and later.

**Description:** The write policy to use when saving records in an Aerospike set. This flag was introduced in Aerospike Connect for Spark 3.0.0 (for Apache Spark 3.x) and 2.8.0 (for Apache Spark 2.4.x).

This flag overrides the specified Spark’s SaveMode setting. For instance, if a user specifies Spark write mode `SaveMode.Overwrite` and sets `aerospike.writepolicy.recordexistsaction` to [`CREATE_ONLY`](https://javadoc.io/doc/com.aerospike/aerospike-client-jdk8/latest/com/aerospike/client/policy/RecordExistsAction.html), then records are written using Java client policy `RecordExistsAction.CREATE_ONLY`.

**Possible values:**

-   `UPDATE` Create or update record. Merge write command bins with existing bins.
-   `UPDATE_ONLY` Update record only. Fail if record does not exist. Merge write command bins with existing bins.
-   `REPLACE` Create or replace record. Delete existing bins not referenced by write command bins.
-   `REPLACE_ONLY` Replace record only. Fail if record does not exist. Delete existing bins not referenced by write command bins.
-   `CREATE_ONLY` Create only. Fail if record exists.

**Default value:** `UPDATE`

**Required?:** No

---

`aerospike.write-set`

**Alias** `aerospike.writeset` is deprecated for versions 5.0.0 and later.

**Description:** The Aerospike set into which records from a Spark Dataframe are to be written.

**Possible values:** N/A

**Default value:** The set specified by `aerospike.read-set`

**Required?:** No

---

#### Example of setting `aerospike.seed-nodes` for loading from one Aerospike cluster and writing the DataFrame into another

```plaintext
dfCluster1=spark.read.format("aerospike").

.option("aerospike.seed-nodes", "cluster1:3000").option("aerospike.read-set", "cluster1").load()

dfCluster1.write.format("aerospike").

.option("aerospike.seed-nodes", "cluster2:3000").option("aerospike.write-set", "cluster2").save()
```

#### Example of setting `aerospike.pushdown-expressions`

How to retrieve records of a set `table` with bin name `col` where content of the bin `col` is divisible by 5.

```plaintext
import com.aerospike.spark.utility.AerospikePushdownExpressions

// col % 5 == 0

// Equivalent: Exp.eq(Exp.mod(Exp.intBin("col"), Exp.`val`(5)), Exp.`val`(0))

val expIntBin=AerospikePushdownExpressions.intBin("col") // id is the name of column

val expMODIntBinEqualToZero=AerospikePushdownExpressions.eq(

  AerospikePushdownExpressions.mod(expIntBin,

    AerospikePushdownExpressions.`val`(5)),

  AerospikePushdownExpressions.`val`(0))

val expMODIntBinToBase64= AerospikePushdownExpressions.build(expMODIntBinEqualToZero).getBase64

val pushDownDF =spark.sqlContext

.read

.format("aerospike")

.option("aerospike.read-set", "table")

.option("aerospike.pushdown-expressions", expMODIntBinToBase64)

.load()
```

### Flexible schemas

The configuration parameter `aerospike.flexible-schema` determines the response of the connector when the data type of a bin in an Aerospike record differs from the data type of the corresponding column in the schema. The schema can be inferred or provided. Spark SQL assumes that the records in a data store (an Aerospike set, in this case) use a single schema. However, as a NoSQL database, Aerospike does not require records in a set to use a single schema. An Aerospike bin (which maps to a column in a Spark DataFrame) within an Aerospike set (which maps to a Spark DataFrame) could contain values that are of multiple data types. The configuration parameter `aerospike.flexible-schema`, when set to `true`, reconciles this incompatibility. For more information, see the section [Flexible schema inference in this notebook](https://nbviewer.jupyter.org/github/aerospike-examples/interactive-notebooks/blob/main/notebooks/spark/AerospikeSparkPython.ipynb#Flexible-schema-inference).

#### Possible values

The configuration parameter `aerospike.flexible-schema` accepts the values `true` and `false`.

`true` If this is the value, and one or more of the data types for columns in a schema do not match the types of the corresponding bins in an Aerospike record, the connector returns NULL values for those columns. **Note:** NULL values due to mismatches are indistinguishable from missing values. Therefore, a client application must treat all NULLs as missing values. To filter out NULLs, use Spark’s `filter()` function, as in this example: `df.filter("gender == NULL").show(false)`

If `false`, and one or more of the data types for columns in a schema do not match the types of the corresponding bins in an Aerospike record, queries fail. You can set to `false` if you have modeled your data in Aerospike to adhere to a strict schema (that is, each record within the set has the same schema).

### Data type mapping

The Spark connector maps Aerospike bin values to Spark SQL types during DataFrame reads as follows:

| Spark SQL type | Aerospike source type | Notes |
| --- | --- | --- |
| `ShortType` | `Long` | Range-checked; throws an error if the value exceeds the `Short` range |
| `IntegerType` | `Long` | Range-checked; throws an error if the value exceeds the `Integer` range |
| `LongType` | `Long` | Also accepts `Timestamp` and `Date` values (converted via epoch milliseconds) |
| `FloatType` | `Double` | Range-checked; throws an error if the value exceeds the `Float` range |
| `DoubleType` | `Double` | See [Floating-point precision](#floating-point-precision) |
| `DateType` | `Long` | Interpreted as epoch milliseconds and converted to a date |
| `TimestampType` | `Long` | Interpreted as epoch milliseconds and converted to a timestamp |
| `BooleanType` | `Boolean` or `Long` | Native `Boolean` values are passed through. `Long` values of `0` and `1` are converted to `false` and `true`. Other `Long` values throw an error. |
| `StringType` | `String` |  |
| `GeoJSONType` | `GeoJSON` | Treated the same as `StringType` |
| `BinaryType` | `Bytes` |  |
| `ByteType` | `Byte` |  |
| `ArrayType` | `List` | Element types are converted recursively |
| `MapType` | `Map` | Key and value types are converted recursively |
| `StructType` | `Map` | Nested fields are converted recursively from a `Map` |

If the connector encounters a type that is not in this table, it throws a `TypeMismatchException`. When you set `aerospike.flexible-schema` to `true`, type conversion errors return `null` instead of throwing an exception. See [Flexible schemas](#flexible-schemas) for details.

::: note
Aerospike does not have a native decimal type. If your source data uses a decimal or fixed-precision type, convert it before writing to Aerospike. The most common approach is to cast decimal values to `DoubleType`, but see [Floating-point precision](#floating-point-precision).
:::

#### Floating-point precision

`DoubleType` in Spark (and the underlying `double` type in Aerospike) uses IEEE 754 double-precision floating point, which provides approximately 15-17 significant decimal digits of precision. The connector rounds values beyond this range.

If your application requires absolute precision, consider storing values as scaled integers instead of doubles. Multiply by a fixed power of 10 before writing (for example, multiply by 1015 for 15 decimal places of precision), store the result as a `LongType`, and convert back when reading.

## Performance

The Spark connector uses the following configuration parameters to tune the connector for optimal performance. These parameters apply to the connector only, not to Aerospike Database or Spark.

`aerospike.key-lookup-batchsize`

**Alias** `aerospike.batchmax` is deprecated for versions 5.0.0 and later.

**Description:** The `aerolookup` function uses Aerospike batch-read requests to read data from an Aerospike database. This property sets the maximum batch size to improve throughput. You must set the value of `aerospike.key-lookup-batchsize` to be less than or equal to the value of `batch-max-requests`. For more information about the `aerolookup` function, see [this tutorial](https://aerospike.com/docs/connectors/spark/tutorials/aerolookup).

**Possible values:** N/A

**Default value:** 5000

**Required?:** No

---

`aerospike.clientpolicy.compression`

**Alias** `aerospike.compression` is deprecated for versions 5.0.0 and later.

**Description:** Enables the compression of data sent between the connector and an Aerospike database.

**Possible values:** `true`, `false`

**Default value:** `false`

**Required?:** No

---

`aerospike.partition-factor`

**Alias** `aerospike.partition.factor` is deprecated for versions 5.0.0 and later.

**Description:** Specifies the number of Spark partitions to map to the 4096 partitions of the selected Aerospike namespace. (A Spark partition is the fundamental unit of parallelism in Spark. Aerospike evenly distributes a namespace’s data across 4096 Aerospike partitions.) Examples:

-   Setting the value to 8 maps 4096 Aerospike partitions to 2^8 or 256 Spark partitions.
-   Setting the value to 15 maps 4096 Aerospike partitions to 2^15 or 32K Spark partitions.
-   Setting the value to 0 maps 4096 Aerospike partitions to 2^0 or 1 Spark partition. In this case, the job is not parallelized.

You should be able to achieve massive parallelization by sizing your Spark cluster to support reads from a large number of partitions. In versions of Aerospike Connect for Spark before 2.2.0, the number of Spark partitions is computed by `4096 >>VALUE`.

**Possible values:** N/A

**Default value:** 8 In versions of Aerospike Connect for Spark before 2.2.0, the default value is 0.

**Required?:** Yes, if you are also using `aerospike.sample-size`.

---

`aerospike.scanpolicy.recordspersecond`

**Alias** `aerospike.recordspersecond` is deprecated for versions 5.0.0 and later.

**Description:** The maximum number of records per second that one compute unit (Spark partition) can request. For example, if a Spark batch read job uses 8 compute units (supposing that the value of `aerospike.partition-factor` is 3) and the value of `aerospike.scanpolicy.recordspersecond` is 100, then the job can request no more than 800 records per second.

::: tip
`aerospike.scanpolicy.recordspersecond` throttles the throughput for scans. The related property `aerospike.write-rate` throttles the throughput for batch reads. We recommend you configure both properties.
:::

**Possible values:** N/A

**Default value:** 0

**Required?:** No

---

`aerospike.sample-size`

**Alias** `aerospike.sample.size` is deprecated for versions 5.0.0 and later.

**Description:** Specifies the number of records to sample from your Aerospike database and write into a Spark DataFrame. This property allows you to avoid loading vast amounts of data from Aerospike into a Spark DataFrame before you can perform an analysis of only a sample of that data. For example, you could sample a small percentage of the data across globally distributed Aerospike clusters and then aggregate the results. You must use this configuration property together with the configuration property `aerospike.partition-factor`, which determines the parallelism in the connector and impacts query latency. Tune it carefully based on available resources, otherwise you may notice performance degradation. See the connector configuration page for more information. For example, if you have 12 vCPUs in the system, a partition factor of 10 or 11 may give the best performance (query latency) for your queries. For higher values, performance may degrade. Based on our testing, for the values of `aerospike.partition-factor` higher than 10, the records read from an Aerospike database are greater than the sample size that you set with `aerospike.sample-size`. Consider also using either Spark’s `sampling()` function or its `limit()` function to ensure that the number of records to your Spark DataFrame equal your desired sample size. Sampling is not random. Each query invocation returns the same set of records. However, the order of records in the result set is randomized. Therefore, if you need random sampling, sample more data that you need and perhaps use the Spark `sampling()` function to randomize the sample. Here is an example snippet of code that shows how to use `aerospike.sample-size`:

```plaintext
val df3=spark.read.format("aerospike")

.option("aerospike.partition-factor","2")

.option("aerospike.read-set", setname)

.option("aerospike.sample-size", "101")

.load()
```

Refer to the notebook tutorials for further details about setting the values of configuration parameters.

**Possible values:** N/A

**Default value:** 0

**Required?:** No

---

`aerospike.write-rate`

**Alias** `aerospike.transaction.rate` is deprecated for versions 5.0.0 and later.

**Description:** This configuration property is available in Aerospike Connect for Apache Spark 3.2.0 and later. It throttles the transactions (reads and/or writes) per second between each Spark partition and the Aerospike clusters. It is not mandatory to use this property in conjunction with the Aerospike Rate Quotas, which ensures that you will not exceed specified read and write rates.

When writing to the Aerospike database, the value of this property is applied to each Spark partition that writes data to Aerospike. For example, if you set the value of `aerospike.write-rate` to 4000 and your Spark application uses 10 partitions for writing, the maximum possible writes per second (WPS) is 4000 \* 10 or 40,000, with a maximum of 4000 per partition. This property is applicable to writes and batch-reads used in the `aerolookup` function and the Spark queries that use batch-read requests.

If the write/batch-read rate exceeds the calculated limit, then the connector blocks until the requested transaction meets or satisfies the set limit.

::: tip
`aerospike.write-rate` throttles the throughput for batch reads. The related property `aerospike.scanpolicy.recordspersecond` throttles the throughput for scans. We recommend you configure both properties.
:::

**Possible values:** positive float values

**Default value:** No rate limits on the transactions.

**Required?:** No

---

`aerospike.write.buffered`

::: note
This flag was discontinued in version 5.0.0. If specified, the connector logs a warning and ignores it. Batch writes are now always used when the Aerospike Database version supports them (6.0 and later).
:::

**Description:** If set to false, [non-batched](https://javadoc.io/static/com.aerospike/aerospike-client/6.1.6/com/aerospike/client/AerospikeClient.html#put-com.aerospike.client.policy.WritePolicy-com.aerospike.client.Key-com.aerospike.client.Bin...-) writes are used. If set to true, [batch](https://aerospike.com/docs/develop/learn/batch) (requires Database 6.0 and later releases) is used for record insertion.

This flag applies to both Apache Spark batch and stream writes. This means you can mix and match buffered writes in streaming and batch applications. This feature was available in Spark connector versions 3.5.0 through 4.5.1.

**Possible values:** `true`, `false`

**Default value:** `true`

**Required?:** No

---

`aerospike.pushdown-expressions`

**Alias** `aerospike.pushdown.expressions` is deprecated for versions 5.0.0 and later.

**Description:** Spark filter class allows limited operators in a predicate to be pushed down to the database, hence the Spark connector is limited in the number of Aerospike expressions that it can generate and pushdown to the database. This property allows you to pushdown Base64 encoded [Aerospike Expressions](https://aerospike.com/docs/develop/expressions) directly to the database, thereby significantly reducing the data movement between Aerospike and the Spark clusters and consequently improving the performance significantly. For the list of supported Exp object and how to compose them, see [Expression generator](https://javadoc.io/doc/com.aerospike/aerospike-client-jdk8/latest/com/aerospike/client/exp/Expression.html) in the API documentation for Aerospike’s Java client library. Here are a few key points to be aware of when using this property:

-   All expressions-related functionality is defined in `com.aerospike.spark.utility.AerospikePushdownExpressions` package.
-   **Do NOT use this property in conjunction with a Spark predicate, which is typically located in the WHERE clause or a filter().**
-   To use this property in conjunction with the `aerolookup` API, specify it in the `SparkConf` object as a key-value pair.

See the [Jupyter Notebook](https://aerospike.com/docs/connectors/spark/tutorials/notebook) for more information on how to push Aerospike expressions down to the database.

**Possible values:** N/A

**Default value:** ""

**Required?:** No

---

`aerospike.clientpolicy.maxconnspernode`

**Alias** `aerospike.client.maxconnspernode` is deprecated for versions 5.0.0 and later.

**Description:** This option was introduced in the connector 4.2.0 release. It sets the underlying [Java client maxConnsPerNode](https://javadoc.io/doc/com.aerospike/aerospike-client-jdk8/latest/com/aerospike/client/policy/ClientPolicy.html#maxConnsPerNode). The Java client uses this flag in sync read/write operations. The Spark connector uses the following synchronous Java client APIs:

-   [`get`](https://javadoc.io/static/com.aerospike/aerospike-client/7.1.0/com/aerospike/client/AerospikeClient.html#get-com.aerospike.client.policy.BatchPolicy-com.aerospike.client.Key:A-) in the `aerolookup` API (in all connector versions) and read queries involving lookup from the Aerospike database.
-   [`put`](https://javadoc.io/static/com.aerospike/aerospike-client/7.1.0/com/aerospike/client/AerospikeClient.html#put-com.aerospike.client.policy.WritePolicy-com.aerospike.client.Key-com.aerospike.client.Bin...-) while writing data into Aerospike database versions prior to 6.0.0.

**Possible values:** N/A

**Default value:** 100

**Required?:** No

---

`aerospike.clientpolicy.minconnspernode`

**Alias** `aerospike.client.minconnspernode` is deprecated for versions 5.0.0 and later.

**Description:** This option was introduced in the connector 4.2.0 release. It sets the underlying [Java client minConnsPerNode](https://javadoc.io/doc/com.aerospike/aerospike-client-jdk8/latest/com/aerospike/client/policy/ClientPolicy.html#minConnsPerNode). The Java client uses this flag in sync read/write commands when the connector uses the Java client synchronous API (see `aerospike.clientpolicy.maxconnspernode` section).

**Possible values:** N/A

**Default value:** 0

**Required?:** No

---

`aerospike.clientpolicy.asyncmaxconnspernode`

**Alias** `aerospike.client.asyncmaxconnspernode` is deprecated for versions 5.0.0 and later.

**Description:** This option was introduced in the connector 4.2.0 release. It sets the underlying [Java client asyncMaxConnsPerNode](https://javadoc.io/doc/com.aerospike/aerospike-client-jdk8/latest/com/aerospike/client/policy/ClientPolicy.html#asyncMaxConnsPerNode). The Java client uses this flag in asynchronous read/write commands when the connector uses the Java client asynchronous APIs. The Spark connector uses the following asynchronous Java client APIs:

-   [`scanPartitions`](https://javadoc.io/static/com.aerospike/aerospike-client/7.1.0/com/aerospike/client/AerospikeClient.html#scanPartitions-com.aerospike.client.async.EventLoop-com.aerospike.client.listener.RecordSequenceListener-com.aerospike.client.policy.ScanPolicy-com.aerospike.client.query.PartitionFilter-java.lang.String-java.lang.String-java.lang.String...-) and [`queryPartitions`](https://javadoc.io/static/com.aerospike/aerospike-client/7.1.0/com/aerospike/client/AerospikeClient.html#queryPartitions-com.aerospike.client.async.EventLoop-com.aerospike.client.listener.RecordSequenceListener-com.aerospike.client.policy.QueryPolicy-com.aerospike.client.query.Statement-com.aerospike.client.query.PartitionFilter-) in scan read queries.
-   [`operate`](https://javadoc.io/static/com.aerospike/aerospike-client/7.1.0/com/aerospike/client/AerospikeClient.html#operate-com.aerospike.client.async.EventLoop-com.aerospike.client.listener.BatchOperateListListener-com.aerospike.client.policy.BatchPolicy-java.util.List-) to write data into Aerospike database versions 6.0+ (introduced in the connector [3.5.0 release](https://aerospike.com/docs/connectors/spark/release-notes#connect-for-spark-350)).

**Possible values:** N/A

**Default value:** -1 (uses the Java client default)

**Required?:** No

---

`aerospike.clientpolicy.asyncminconnspernode`

**Alias** `aerospike.client.asyncminconnspernode` is deprecated for versions 5.0.0 and later.

**Description:** This option was introduced in the connector 4.2.0 release. It sets the underlying [Java client asyncMinConnsPerNode](https://javadoc.io/doc/com.aerospike/aerospike-client-jdk8/latest/com/aerospike/client/policy/ClientPolicy.html#asyncMinConnsPerNode). The Java client uses this flag in asynchronous read/write commands when the connector uses the Java client asynchronous APIs (see `aerospike.clientpolicy.maxconnspernode` section).

**Possible values:** N/A

**Default value:** 0

**Required?:** No

---

`aerospike.writepolicy.durabledelete`

**Alias** `aerospike.client.durabledelete` is deprecated for versions 5.0.0 and later.

**Description:** This option was introduced in the connector 4.3.0 release. It sets the underlying [Java client BatchWritePolicy durableDelete](https://javadoc.io/static/com.aerospike/aerospike-client/7.1.0/com/aerospike/client/policy/BatchWritePolicy.html#durableDelete) or [Java Client WritePolicy durableDelete](https://javadoc.io/static/com.aerospike/aerospike-client/7.1.0/com/aerospike/client/policy/WritePolicy.html#durableDelete). BatchWritePolicy sets `durableDelete` when batch writes are used in Aerospike Database 6.0 and later.

**Possible values:** true, false

**Default value:** false

**Required?:** No

---

## Authentication

`aerospike.clientpolicy.password`

**Alias** `aerospike.password` is deprecated for versions 5.0.0 and later.

**Description:** Password for authentication. Leave null for clusters running without restricted access.

**Default value:** null

**Required?:** No

---

`aerospike.tls.allowedpeerNames`

**Description:** The valid peer names that are allowed in the TLS client certificate, if mutual authentication is required.

**Default value:** N/A

**Required?:** No

---

`aerospike.client.policy.authmode`

**Alias** `aerospike.tls.authmode` is deprecated for versions 5.0.0 and later.

**Description:** [Supported authentication modes](https://javadoc.io/doc/com.aerospike/aerospike-client-jdk8/latest/com/aerospike/client/policy/AuthMode.html).

**Possible values:**

| Value | Description |
| --- | --- |
| 0 | Use internal authentication only. The hashed password is stored on the server. Do not send clear password. This is the default. |
| 1 | Use external authentication (such as LDAP). Specific external authentication is configured on the server. If TLS defined, send clear password on node login using TLS. Throw exception if TLS is not defined. |
| 2 | Use external authentication (such as LDAP). Specific external authentication is configured on the server. Send clear password on node login whether TLS is defined. Use only for testing because it does not provide secure authentication. |
| 3 | Authentication and authorization based on a certificate (PKI). No user name or password needs to be configured. Requires TLS and a client certificate. Requires Database 5.7.0 or later. Supported since the release of connector 4.1.0 |

**Default value:** 0

**Required?:** No

---

`aerospike.tls.ciphers`

**Description:** Comma-separated values of TLS ciphers. Available [cipher names](https://docs.oracle.com/javase%2F7%2Fdocs%2Fapi%2F%2F/javax/net/ssl/SSLSocket.html#getSupportedCipherSuites\(\)). A value of null allows the default ciphers that are defined by the JVM.

**Default value:** null

**Required?:** No

---

`aerospike.tls-enabled`

**Alias** `aerospike.tls.enabletls` is deprecated for versions 5.0.0 and later.

**Description:** Boolean flag for whether to use TLS client-server communication. If the value is `true`, TLS is enabled for communication between Aerospike and the connector.

**Default value:** `false`

**Required?:** No

---

`aerospike.clientpolicy.tls.forloginonly`

**Alias** `aerospike.tls.forloginonly` is deprecated for versions 5.0.0 and later.

**Description:** A value of `true` specifies to use TLS connections only for login authentication; all other communication with the server use non-TLS connections. A value of `false` means that TLS connections are used for all communications with the server.

**Default value:** `false`

**Required?:** No

---

`aerospike.clientpolicy.tls.keystore-store-file`

**Alias** `aerospike.tls.keystore-store-file` is deprecated for versions 5.0.0 and later.

**Description:** Path of the keystore store file

**Default value:** N/A

**Required?:** Yes, if you are using a keystore.

---

`aerospike.clientpolicy.tls.keystore-key-password-file`

**Alias** `aerospike.tls.keystore-key-password-file` is deprecated for versions 5.0.0 and later.

**Description:** Path of the keystore key password file

**Default value:** N/A

**Required?:** Yes, if you are using a keystore.

---

`aerospike.clientpolicy.tls.keystore-store-password-file`

**Alias** `aerospike.tls.keystore-store-password-file` is deprecated for versions 5.0.0 and later.

**Description:** Path of the keystore store password file

**Default value:** N/A

**Required?:** Yes, if you are using a keystore.

---

`aerospike.clientpolicy.tls.keystore-store-type`

**Alias** `aerospike.tls.keystore-store-type` is deprecated for versions 5.0.0 and later.

**Description:** The connector supports both the proprietary Java Keystore format (“JKS”) as well as the “PKCS12” format, based on the RSA PKCS12 Personal Information Exchange Syntax Standard. The default keystore format is JKS. One of the differences between the two formats is that JKS protects each private key with its individual password, while also protecting the integrity of the entire keystore with a (possibly different) password. A PKCS12 keystore, on the other hand, only uses a single password for the entire keystore. For more information, see “[Setting Up TLS Keystores for Aerospike Connect](https://aerospike.com/docs/connectors/streaming/common/keystores)”.

**Default value:** `JKS`

**Required?:** Yes, if you are using a keystore.

---

`aerospike.clientpolicy.tls.protocols`

**Alias** `aerospike.tls.protocols` is deprecated for versions 5.0.0 and later.

**Description:** Comma-separated list of TLS protocols. All possible values are `TLSv1`, `TLSv1.1`, `TLSv1.2`

**Default value:** `TLSv1.2`

**Required?:** Yes, if you are using TLS.

---

`aerospike.clientpolicy.tls.revokecertificates`

**Alias** `aerospike.tls.revokecertificates` is deprecated for versions 5.0.0 and later.

**Description:** Comma-separated list of revoked certificates in long (arabic numeral) format.

**Default value:** N/A

**Required?:** No

---

`tls-name`

**Alias** `aerospike.tls.tlsname` is deprecated for versions 5.0.0 and later.

**Description:** The connector uses the `tls-name` to authenticate each TLS socket connection against a server node, based on the certificate presented by the Aerospike Database node during the initial connection handshake. `tls-name` for a node is typically the node’s hostname.

**Default value:** N/A

**Required?:** No

---

`aerospike.clientpolicy.tls.truststore-store-file`

**Alias** `aerospike.tls.truststore-store-file` is deprecated for versions 5.0.0 and later.

**Description:** Path of the truststore store file

**Default value:** N/A

**Required?:** Yes, if you are using a truststore.

---

`aerospike.clientpolicy.tls.truststore-key-password-file`

**Alias** `aerospike.tls.truststore-key-password-file` is deprecated for versions 5.0.0 and later.

**Description:** Path of the truststore key password file

**Default value:** N/A

**Required?:** Yes, if you are using a truststore.

---

`aerospike.clientpolicy.tls.truststore-store-password-file`

**Alias** `aerospike.tls.truststore-store-password-file` is deprecated for versions 5.0.0 and later.

**Description:** Path of the truststore store password file

**Default value:** N/A

**Required?:** Yes, if you are using a truststore.

---

`aerospike.clientpolicy.tls.truststore-store-type`

**Alias** `aerospike.tls.truststore-store-type` is deprecated for versions 5.0.0 and later.

**Description:** The truststore store type.

**Default value:** `JKS`

**Required?:** Yes, if you are using a truststore.

---

`aerospike.clientpolicy.user`

**Alias** `aerospike.user` is deprecated for versions 5.0.0 and later.

**Description:** User for authentication. Leave null for clusters running without restricted access.

**Default value:** null

**Required?:** No

---

## Secondary index

`aerospike.sindex-enable`

**Alias** `aerospike.sindex.enable` is deprecated for versions 5.0.0 and later.

**Description:** Boolean/String flag to enable secondary index query.

**Possible values:** `true`, `false`

**Default value:** `true`

**Required?:** No

---

`aerospike.sindex`

**Description:** secondary index to be used for query evaluation. If not specified, the connector will select an appropriate index to match the order of Spark columns in the `where` clause, provided `aerospike.sindex-enable` is set.

User may invoke `sindexList(namepsace)` API to list all the secondary indices defined in a namespace. This API assumes that `SparkSession` has the parameters required to create a connection to the database set, including `aerospike.seed-nodes`, `aerospike.namespace` and other configurations. See the notebooks for a complete example.

```scala
//Scala

import com.aerospike.spark._

PythonUtil.sindexList("test") //namespace = "test"
```

```python
#python

scala_py_util = sc._jvm.com.aerospike.spark.PythonUtil #sc is sparkContext

print(scala_py_util.sindexList("test")) #namespace = "test"
```

**Possible values:** Any secondary index

**Default value:** ""

**Required?:** No

---

`aerospike.sindex-filter`

**Alias** `aerospike.sindex.filter` is deprecated for versions 5.0.0 and later.

**Description:** JSON representation of the filter. If set, the constructed filter will be used to query the secondary index. `aerospike.sindex` must be set with `aerospike.sindex-filter` flag. See the [Secondary Index Support](https://aerospike.com/docs/connectors/spark/spark-sindex/) for information on setting different types of filters.

**Possible values:** `contains`, `range` and `equal` filter in JSON format

**Default value:** ""

**Required?:** No

## Exponential backoff retry

**Connector 3.3.0 and later**

When certain error conditions occur as the connector interacts with your Aerospike cluster, the connector can “back off” from the server. “Backing off” means not only to retry the actions that led to the error, but to retry them at exponentially-increasing intervals. The duration of the interval before the first attempt is specified by the configuration property `aerospike.retry-initial-millis`. If the database cannot service the request because it is busy, the connector attempts the same action after exponentially longer intervals. To compute the length of each successive interval, the connector multiplies the duration of the current interval by the value of the configuration parameter `aerospike.retry-multiplier`.

For example, if the initial wait time is 1s (1000 milliseconds) and the multiplier is 2, the retries are attempted after 1s, 2s, 4s, 8s, 16s, 32s, and so on. The connector retries the action until the database can service the request, or until the connector reaches the maximum number of retries allowed, which you can specify with `aerospike.retry-max-attempts`.

Violation of a rate quota is the most important error conditions that prompts the connector to back off from the server. The other error conditions are [internal error](https://javadoc.io/doc/com.aerospike/aerospike-client-jdk8/latest/constant-values.html#com.aerospike.client.ResultCode.NOT_AUTHENTICATED) conditions. You can specify retry specific error codes using `aerospike.retry-errorcodes`.

`aerospike.retry-initial-millis`

**Alias** `aerospike.retry.initialmillis` is deprecated for versions 5.0.0 and later.

**Description:** The time to wait (in milliseconds) before retrying for the first time the action that led to the error condition. If the error condition persists after the initial retry, subsequent retries are attempted after intervals that become exponentially longer.

**Default value:** `1000`

**Required?:** No

---

`aerospike.retry-max-attempts`

**Alias** `aerospike.retry.maxattempts` is deprecated for versions 5.0.0 and later.

**Description:** The maximum number of times to retry an action that led to an error condition. A value of 0 prevents the connector from retrying.

**Default value:** `0`

**Required?:** No

---

`aerospike.retry-multiplier`

**Description:** The integer by which to multiply the duration of the current wait interval to determine the duration of the next wait interval.

**Default value:** `2`

**Required?:** No

---

`aerospike.retry-errorcodes`

**Alias** `aerospike.retry.errorcodes` is deprecated for versions 5.0.0 and later.

**Description:** String comma-separated [error codes](https://javadoc.io/doc/com.aerospike/aerospike-client-jdk8/latest/constant-values.html#com.aerospike.client.ResultCode.NOT_AUTHENTICATED) to trigger exponential backoff retry.

**Default value:** `83,75,-7,-9,-12,-8`

**Required?:** No

---