Configure Kafka producer properties

The producer-props section of the /etc/aerospike-kafka-outbound/aerospike-kafka-outbound.yml lets you specify a map of the Kafka producer properties. You can specify all of the properties in ProducerConfig in your map. When the Kafka producer runs, it is initialized with these properties.

Caution

Key and value serializer properties specified in the config are ignored and are always overwritten with a BytesSerializer.

Examples

Here is a sample map:

producer-props:
  bootstrap.servers:
    - 192.168.5.20:9092
    - 192.168.5.30:9092
  acks: all
  retries: 0

Kafka uses the Java Authentication and Authorization Service (JAAS) for SASL configuration. You must provide JAAS configuration properties for any SASL authentication mechanisms that your Apache Kafka cluster uses. For more information about SASL and its use by Apache Kafka, see Authentication with SASL using JAAS in the Apache Kafka documentation.

Here is a sample map that shows a configuration for Simple Authentication and Security Layer (SASL), which Apache Kafka supports:

producer-props:
  bootstrap.servers:
    - kafkabroker-usw2-1:9092
    - kafkabroker-use2-1:9092
  ssl.truststore.location: /var/ssl/private/client.truststore.jks
  security.protocol: SASL_SSL
  sasl.mechanism: SCRAM-SHA-256
  sasl.jaas.config: org.apache.kafka.common.security.scram.ScramLoginModule required username="kafkaclient" password="kafkaclient-secret";

Aerospike Kafka source (outbound) connector can also be used with the AWS MSK. Here is a sample map that shows a configuration for the AWS MSK with IAM authentication:

Note

To use AWS MSK with IAM authentication, download and put aws-msk-iam-auth-{LATEST_VERSION}-all.jar under /opt/aerospike-kafka-outbound/usr-lib directory. Also, make sure that the connector can be authorized by the Default credentials provider chain in the AWS SDK for Java.

producer-props:
    bootstrap.servers:
      - b-1-public.mycluster.1a1bcd.c3.kafka.ap-south-1.amazonaws.com:9198
      - b-2-public.mycluster.1a1bcd.c3.kafka.ap-south-1.amazonaws.com:9198
      - b-3-public.mycluster.1a1bcd.c3.kafka.ap-south-1.amazonaws.com:9198
    security.protocol: SASL_SSL
    sasl.mechanism: AWS_MSK_IAM
    sasl.jaas.config: software.amazon.msk.auth.iam.IAMLoginModule required;
    sasl.client.callback.handler.class: software.amazon.msk.auth.iam.IAMClientCallbackHandler