Upgrade Operator 2.5.0 to 3.0.0 from OperatorHub.io

This is the procedure to upgrade the Operator, installed from OperatorHub.io, from version 2.5.0 to version 3.0.0.

Verify that version 3.0.0 of the operator is available

Verify that the following command lists 3.0.0 as the current CSV version:

kubectl get packagemanifests  -l catalog=operatorhubio-catalog,provider=Aerospike -o yaml | grep currentCSV:

The following appears:

    - currentCSV: aerospike-kubernetes-operator.v2.5.0
    - currentCSV: aerospike-kubernetes-operator.v3.0.0

Upgrade the operator

Based on the installPlanApproval mode, the upgrade of the Operator can be one of

• Automatic (Not Supported) - the default when installing the Operator from OperatorHub.io.
• Manual - if the OperatorHub.io subscription has been edited to use Manual approval.

Automatic

Operator 3.0.0 doesn't support Automatic upgrade because the Operator release channel has been changed from alpha to stable. Refer Manual section below.

Manual

To upgrade Operator to 3.0.0 version, edit the subscription to change the channel from alpha to stable:

kubectl patch subscriptions -n operators --type merge --patch '{"spec":{"channel":"stable"}}' $(kubectl get subscriptions -n operators | grep "aerospike-kubernetes-operator" | cut -f 1 -d " ")

If the Operator's subscription is set for Automatic approval (Default), then Operator will automatically upgrade to latest version. You can skip ahead to Verify Operator upgrade step.

If the OperatorHub.io subscription has been edited to use Manual approval, then you need to approve the upgrade as follows:

Verify that the InstallPlan for version 3.0.0 has been created.

kubectl get installplan -n operators | grep aerospike

Sample output with an InstallPlan for version 3.0.0.

NAME            CSV                                    APPROVAL   APPROVED
install-2tg7p   aerospike-kubernetes-operator.v3.0.0   Manual     false
install-fn297   aerospike-kubernetes-operator.v2.5.0   Manual     true

The upgrade is not applied, since the approved status is false.

To approve the upgrade, set the approved field in the InstallPlan to true:

kubectl patch installplan -n operators --type merge --patch '{"spec":{"approved":true}}'  $(kubectl get installplan -n operators | grep "aerospike-kubernetes-operator.v3.0.0" | cut -f 1 -d " ")

Verify that the Operator is upgraded

Run the following command

kubectl get csv -n operators | grep aerospike

The operator upgrade can take some time. The CSV for version 3.0.0 goes through phases like Pending, Installing, InstallReady and finally on Succeeded.

Sample output on success:

NAME                                   DISPLAY                         VERSION   REPLACES                               PHASE
aerospike-kubernetes-operator.v3.0.0   Aerospike Kubernetes Operator   3.0.0     aerospike-kubernetes-operator.v2.5.0   Succeeded

Check Operator Logs

The Operator runs as two replicas by default, for higher availability. Run the following command to follow the logs for the Operator pods.

kubectl -n operators logs -f deployment/aerospike-operator-controller-manager manager

Output:

2023-08-01T09:07:02Z    INFO    setup   legacy OLM < 0.17 directory is present - initializing webhook server
2023-08-01T09:07:03Z    INFO    controller-runtime.metrics  Metrics server is starting to listen    {"addr": "127.0.0.1:8080"}
2023-08-01T09:07:03Z    INFO    setup   Init aerospike-server config schemas
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "4.3.0"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "4.5.2"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "5.6.0"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "4.5.0"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "5.2.0"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "5.4.0"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "4.0.0"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "4.7.0"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "6.0.0"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "5.7.0"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "4.1.0"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "5.1.0"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "4.5.1"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "4.6.0"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "5.0.0"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "6.1.0"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "6.2.0"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "6.4.0"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "6.3.0"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "4.2.0"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "4.5.3"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "5.5.0"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "5.3.0"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "4.3.1"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "4.4.0"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "4.8.0"}
2023-08-01T09:07:03Z    DEBUG   schema-map  Config schema added {"version": "4.9.0"}
2023-08-01T09:07:03Z    INFO    aerospikecluster-resource   Registering mutating webhook to the webhook server
2023-08-01T09:07:03Z    INFO    controller-runtime.webhook  Registering webhook {"path": "/mutate-asdb-aerospike-com-v1-aerospikecluster"}
2023-08-01T09:07:03Z    INFO    controller-runtime.builder  skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn't called   {"GVK": "asdb.aerospike.com/v1, Kind=AerospikeCluster"}
2023-08-01T09:07:03Z    INFO    controller-runtime.builder  Registering a validating webhook    {"GVK": "asdb.aerospike.com/v1, Kind=AerospikeCluster", "path": "/validate-asdb-aerospike-com-v1-aerospikecluster"}
2023-08-01T09:07:03Z    INFO    controller-runtime.webhook  Registering webhook {"path": "/validate-asdb-aerospike-com-v1-aerospikecluster"}
2023-08-01T09:07:03Z    INFO    setup   Starting manager
2023-08-01T09:07:03Z    INFO    controller-runtime.webhook.webhooks Starting webhook server
2023-08-01T09:07:03Z    INFO    Starting server {"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"}
2023-08-01T09:07:03Z    INFO    controller-runtime.certwatcher  Updated current TLS certificate
2023-08-01T09:07:03Z    INFO    Starting server {"kind": "health probe", "addr": "[::]:8081"}
I0801 09:07:03.213295       1 leaderelection.go:248] attempting to acquire leader lease operators/96242fdf.aerospike.com...
2023-08-01T09:07:03Z    INFO    controller-runtime.webhook  Serving webhook server  {"host": "", "port": 9443}
2023-08-01T09:07:03Z    INFO    controller-runtime.certwatcher  Starting certificate watcher

Remove deprecated v1beta1 version from AerospikeCluster CustomResourceDefinition:

Operator 3.0.0 introduces v1 API and deprecates v1beta1 API. Operator automatically migrates all existing Custom Resource objects to latest v1 version. So, remove v1beta1 from the CustomResourceDefinition status.storedVersions field.

kubectl patch customresourcedefinitions aerospikeclusters.asdb.aerospike.com --subresource='status' --type='merge' -p '{"status":{"storedVersions":["v1"]}}'

note

The kubectl flag --subresource flag is available starting from kubectl version v1.24. Update kubectl to version 1.24 or later for patching the CRD status.

Grant RBAC permissions to non-aerospike Kubernetes namespaces

caution

There is a known issue with upgrading from version 2.5.0 to 3.0.0. This upgrade scenario revokes the RBAC privileges required to run Aerospike clusters.

If you are upgrading from 2.5.0 to 3.0.0, follow these instructions to restore the required RBAC privileges for each Kubernetes namespace where Aerospike clusters are running.