Logging

A logging interface is provided to handle log messages generated by the client. Most log messages are generated from background cluster tend thread(s) created for each client instance. Messages include node additions/removal and any errors when retrieving node status, peers, partition maps and racks. This information is critical to debug command failures resulting from cluster tend errors.

Enable Client Log

The log is global so it only needs to enabled once at application launch before instantiating AerospikeClient. The client provides a simple stdout log implementation that can be used for command line applications:

Log.SetCallbackStandard()

A custom console log with additional context can be created by:

public class MyConsole
{
    public MyConsole()
    {
        Log.SetContextCallback(LogCallback);
    }

    public void LogCallback(Log.Context context, Log.Level level, String message)
    {
        // Write log messages to the appropriate place.
    }
}

MyConsole console = new MyConsole();

A legacy custom console log without additional context can be created by:

public class MyConsole
{
    public MyConsole()
    {
        Log.SetCallback(LogCallback);
    }

    public void LogCallback(Log.Level level, String message)
    {
        // Write log messages to the appropriate place.
    }
}

MyConsole console = new MyConsole();

Disable Client Log

The client log can be disabled by:

Log.Disable();

Log Level

Client log messages can be filtered by log level.

ERROR Serious errors that require immediate attention.
WARN Warnings regarding cluster tend errors that might lead to command failures.
INFO Informational cluster changes including adding a node and removing a node from the cluster.
DEBUG Extra debug data.

Each level includes the levels above it, so INFO includes INFO, WARN and ERROR messages. Example:

Log.SetLevel(Log.Level.INFO);

Log Messages

Cluster Startup

These log messages occur when Aerospike client is initialized.

Log Level	Message	Description
WARN	`Seed <seed> failed: <message>`	The client failed to connect to a seed on the cluster startup. The next seed, if any, is tried instead. The seed is a hostname or an IP address.
DEBUG	`Add seed <seed>`	The client is about to add a seed to the list of seeds. This happens at client startup when the seed peers are added as new seeds.

Cluster Tend

Cluster tend log message occur in the cluster tend thread that runs in the background. There is one cluster tend thread for each Aerospike client instance. This thread periodically (default once per second) polls all nodes in the cluster for status, peers, partition maps, and racks. The thread also balances connections according to min or max connection configuration and performs node logins when user authentication is enabled.

Log Level	Message	Description
WARN	`Add node <host> failed: <message>`	The cluster tend could not add a node to the cluster due to an unexpected exception.
WARN	`Node remove mismatch. Expected <node count> Received <node count>`	Each node is contacted periodically (it defaults once every second) for cluster status. If refresh fails, it is usually due to socket error or a timeout (defaults to one second).
WARN	`Node <node> refresh failed: <message>`	When node(s) are removed, the client calculates the new expected cluster size. If the expected cluster size is not the actual cluster size after the node removal, the client logs this error. It could happen when a node is scheduled to be removed that was previously removed.
WARN	`Node restart failed: <node> <message>`	The cluster tend detected a quick node restart and the client failed to properly handle the restart. The node login or connection balancing failed.
WARN	`Peer node <node name> is different than actual node <node name> for host <host>`	The cluster tend received a new peer from a node, but the peer’s name is different from the node name that was contacted from the peer address.
WARN	`Cluster tend failed: <message>`	An unexpected error occurred in the cluster tend thread.
INFO	`Add node <node>`	The cluster tend is about to add a new node to the client cluster view.
INFO	`Login to <node>`	User authentication is enabled and the client needs to re-login after a node restart or an expired session token was detected. The new login renews a session token that is used in new connections.
INFO	`Remove node <node>`	The cluster tend is about to remove a node from the client cluster view.
INFO	`Quick node restart detected: node=<node> oldgen=<gen> newgen=<gen>`	The cluster tend is about to remove a node from the client cluster view.
INFO	`Skip orphan node: <node>`	The cluster tend rejected a new orphan node that claimed ownership of all partitions when the cluster size is greater than one.
INFO	`Invalid address <IP address>. access-address is not configured on server.`	A valid, reachable IP address was not found from a node peers. The address is probably internal to the cloud because the server “access-address” is not configured.
INFO	`Namespace <namespace> replication factor changed from <repl factor> to <repl factor>`	The cluster tend received a new partition map from a node that has a different replication factor for a namespace that already exists.
INFO	`<node> regime(<regime>) < old regime(<regime>)`	The cluster tend rejected a node claim of ownership of a partition due to a regime mismatch. This message is only generated once per node partition map parse to avoid spamming the log.
DEBUG	`Address <IP address> <port> failed: <message>`	The cluster tend attempted to validate a new node, but the node IP address is not reachable.
DEBUG	`Update partition map for node <node>`	The cluster tend is retrieving new partition maps from the specified node. Partition maps include the node partition ownership for each namespace. There are a fixed number of partitions (4096) per namespace. Ownership is specified for each replica level within a partition. The number of replica levels corresponds to the namespace replication factor.
DEBUG	`Update racks for node <node>`	The cluster tend is retrieving new racks from the specified node. This occurs only when the client is configured to be rack-aware.
DEBUG	`Failed to create connection: <message>`	The cluster tend attempted to create a sync connection due to the min sync connections configuration and the attempt failed with an error message.
DEBUG	`Node <node> failed to create connection: <message>`	The cluster tend attempted to create an async connection due to the min async connections configuration and the attempt failed with the node IP address/port and an error message.

TLS Connection

These log messages occur when opening a TLS connection.

Log Level	Message	Description
DEBUG	`Invalid certificate policy error: <SslPolicyErrors>`	The client attempted to open a TLS connection and found a problem with the certificate. The error is specified in SslPolicyErrors.
DEBUG	`Invalid certificate serial number: <certificate serial number>`	The client attempted to open a TLS connection and the certificate serial number is invalid. The serial number is display in hexadecimal format.
DEBUG	`Invalid certificate, tlsName not found: <name>`	The client attempted to open a TLS connection and the expected TLS name was not found in the certificate.

Query

These log messages occur when executing a query command.

Log Level	Message	Description
DEBUG	`ResultSet <task id> both 'add' and 'take' failed on abort`	An aggregation query was aborted, but the end marker can’t be added to the result set.

Command

These log messages occur when executing a sync command.

Log Level	Message	Description
DEBUG	`Thread <id> allocate buffer on heap <size>`	A client command requires a buffer size that is greater than the thread local cutoff (128 KB). When this occurs, a new buffer is allocated for that specific command and discarded when the command completes.
DEBUG	`Thread <id> resize buffer to <size>`	A client command requires a buffer size that is greater than the current thread local buffer size, but less than the thread local cutoff (128 KB). When this occurs, the thread local buffer is resized to the new size.

Async Command

These log messages occur when executing an async command.

Log Level	Message	Description
ERROR	`Unexpected exception from OnRecord(): <message>`	When calling async batch listener OnRecord(), the user provided method threw an exception. Avoid this message by catching all exceptions in async listeners so they are not propagated back to the client.
WARN	`OnSuccess() error: <message>`	When calling async listener OnSuccess(), the user provided method threw an exception. Avoid this message by catching all exceptions in async listeners so they are not propagated back to the client.
WARN	`OnFailure() error: <message>`	When calling async listener OnFailure(), the user provided method threw an exception. Avoid this message by catching all exceptions in async listeners so they are not propagated back to the client.
WARN	`AsyncCommand finished with unexpected return status: <status>`	An async command finished, but had already been aborted (usually by the timeout thread) and the abort status is unknown and unexpected.
WARN	`AsyncCommand unexpected return status: <status>`	An async command received an error, but had already been aborted (usually by the timeout thread) and the abort status is unknown and unexpected.
WARN	`AsyncTimeoutQueue error: <message>`	The async timeout thread received an unexpected exception processing an async command timeout.