Upgrading XDR to version 3.8
In version 3.8, Aerospike re-architected its XDR component and merged the previously separate asxdr daemon into the main asd process. In addition, shipping performance is dramatically improved through the use of pipelined shipping. See the 3.8 release details page for the full list of features and enhancements introduced in XDR as part of Aerospike 3.8.
This page describes the required steps to upgrade from a previous version of XDR to version 3.8.
For one way shipping topologies, the following steps can be executed first at the source cluster or at the destination cluster(s). It is good practice, though, to first upgrade the destination cluster(s). As some of the statistic names have changed, the AMC package should be upgraded to version 3.6.8.1 or above. Contact Aerospike Support if you need clarification on any of the following steps.
Preparation steps
The following points are optional but recommended for most production setups.
Verify that the cluster isn't migrating data. If it is migrating, allow the cluster to finish before proceeding. Use the following command to verify migrations are complete:
asadm -e "enable; asinfo -v statistics -l " | grep migrate_progress
Migrations are complete when all
migrate_progress_recv
andmigrate_progress_send
values are 0.If there are no active migrations, backup the cluster. For each namespace run:
asbackup --host <nodeIP> --directory <path/to/backup/namespace> --namespace <namespace-name>
Save the current configuration file on each node. Assuming default location:
sudo cp /etc/aerospike/aerospike.conf /etc/aerospike/aerospike.conf.bac
Update the configuration file on each node
A few of the previous required configuration parameters are not necessary anymore:
Having the xdr daemon part of the asd process obsoletes the following configuration parameters:
xdr-namedpipe-path
- not required anymore, xdr in same process as asd.
xdr-local-node-port
- not required anymore, xdr in same process as asd.
xdr-pidfile
- not required anymore, xdr in same process as asd.
xdr-errorlog-path
- see point 3. below.
xdr-info-port
- can still be used, see point 4. below.
stop-writes-noxdr
- not required anymore, xdr in same process as asd.The following configuration parameters have been deprecated:
xdr-check-data-before-delete
- always on and built in version 3.8.
xdr-read-batch-size
- digests picked up and dispatched to specific threads (xdr-read-threads).
xdr-hotkey-maxskip
- new xdr-hotkey-time-ms introduced to control hotkey updates interval.
xdr-max-recs-inflight
- new tps based throttling through xdr-max-ship-throughput.
xdr-forward-with-gencheck
- should not be used as xdr generation check will break on delete/createContact Aerospike Support if you had any of the above configuration parameters defined in your configuration file.
By default, the XDR log messages are now merged with the main log file. If desired, those can still be separated, by updating the logging configuration as follows:
logging {
file /var/log/aerospike/aerospike.log {
context any info
context xdr critical
context cf:rbuffer critical
}
file /var/log/aerospike/asxdr.log {
context any critical
context xdr info
context cf:rbuffer info
}
}The XDR configuration and statistics are now accessible through the same port as the asd process (default 3000). For backward compatibility with potential monitoring and management tools, the
xdr-info-port
can still be configured and will return the same results as the asd info port.The
xdr-digestlog-path
(which used to bedigestlog-path
) andenable-xdr
are the only configuration parameters that are mandatory in the xdr stanza, along with the datacenter substanza(s). Here is a simple example of a basic xdr stanza:xdr {
enable-xdr true
xdr-digestlog-path /opt/aerospike/xdr/digestlog 100G
datacenter DC1 {
dc-node-address-port xx.xx.xx.xx 3000
dc-node-address-port yy.yy.yy.yy 3000
dc-node-address-port zz.zz.zz.zz 3000
}
}In lieu of
dc-int-ext-ipmap
configuration, which needed to be done on every source cluster, we introducedalternate-address
, which is configured on each of the destination node.
Destination Node Config:network {
service {
address any
port 3000
access-address aa.bb.cc.dd
alternate-address xx.xx.xx.xx
}
}If
alternate-address
is set, the source cluster has to specifydc-use-alternate-services true
in order to use the alternate IP address to connect to the node (instead of services).
Source Node Config:xdr {
...
datacenter DC1 {
dc-node-address-port xx.xx.xx.xx 3000
...
dc-use-alternate-services true
}
}Other compatibility call-outs:
Local reads before shipping are always internal single-record reads in this new version. Therefore, the xdr initiated reads will not show up as read transactions. The xdr initiated read transactions can be tracked through the
stat_read_reqs_xdr
metric.The "noresume" and "resume-nofailover" functionality is obsolete on asd service restart. By default, xdr will resume with failover.
For more information about latest configuration parameters, refer to the Configuration Reference.
Execute steps 3 to 6 on each node, waiting for the node to join the cluster, and, depending on your requirements, waiting for migrations to complete, before proceeding to the next node.
Stop the asd and asxdr daemons
$ sudo service aerospike stop
Stopping aerospike: [ OK ]
$ sudo service aerospike_xdr stop
Stopping aerospike XDR: [ OK ]
Verify that the services are stopped:
$ sudo service aerospike status
asd is stopped
$ sudo service aerospike_xdr status
asxdr is stopped
Install Aerospike version 3.8
Locate the asinstall script in the extracted media. Run the following command to upgrade the node:
$ sudo ./asinstall
Start the new asd daemon
Start Aerospike and verify that it starts:
$ sudo service aerospike start
Starting and checking aerospike: [ OK ]
$ sudo service aerospike status
asd (pid 4389) is running...
During the rolling upgrade, following messages may be seen in the logs of nodes yet to be upgraded:
WARNING (xdr): (xdr_serverside.c::134) Got an unknown XDR message
WARNING (xdr): (xdr_serverside.c::134) Got an unknown XDR message
WARNING (xdr): (xdr_serverside.c::134) Got an unknown XDR message
WARNING (xdr): (xdr_serverside.c::134) Got an unknown XDR message
Those messages would appear during migrations if there was a bit of lag. They indicate that a node running the upgraded version is trying to relog a digest against a node that has not been upgraded yet. During migrations, master partitions ownership would change, requiring digests to be relogged against the new owner of the partition. This type of relogging messages is not supported in previous XDR versions. This message should disappear after migrations complete and the digest corresponding to the migrations timeframe have been processed.
Validate the installation
Verify in the logs that after restart each node is connecting to the correct destination and that it is shipping records (if there are records to be shipped). Check the logs for any unexpected warning messages.
Contact Aerospike Support for any questions or issues encountered during this upgrade process.