Troubleshooting Aerospike on Kubernetes
Pods stuck in pending state
After an Aerospike cluster has been created or updated if the pods are stuck with “Pending” status like so:
Output:
Terminal window
Describe the pod to find the reason for scheduling failure:
kubectl -n aerospike describe pod aerocluster-0-0Under the events section you will find the reason for the pod not being scheduled. For example:
Output
Terminal window
Possible reasons are
- Storage class incorrect or not created. See persistent storage configuration for details.
- 1 node(s) didn’t match Pod’s node affinity - Invalid zone, region, racklabel, or other parameter for the rack configured for this pod.
- Insufficient resources, CPU or memory available to schedule more pods.
Pods keep crashing
After an Aerospike cluster has been created or updated if the pods are stuck with “Error” or “CrashLoopBackOff” status like so:
Output:
Terminal window
Check the following logs to see if pod initialization failed or the Aerospike Server stopped.
Init logs:
kubectl -n aerospike logs aerocluster-0-0 -c aerospike-initServer logs:
kubectl -n aerospike logs aerocluster-0-0 -c aerospike-serverPossible reasons are
- Missing or incorrect feature key file - Fix by deleting the Aerospike secret and recreating it with correct feature key file. See Aerospike secrets for details.
- Bad Aerospike configuration - AKO tries to validate the configuration before applying it to the cluster. However, it’s still possible to misconfigure the Aerospike server. The offending parameter is logged in the server logs and should be fixed and applied again to the cluster. See Aerospike configuration change for details.
Error connecting to the cluster from outside Kubernetes
If the cluster runs fine as verified by the pod status and asadm (see connecting with asadm), Ensure that firewall allows inbound traffic to the Kubernetes cluster for the Aerospike ports. See Port access for details.
Events
Kubernetes events are generated on errors, resource state changes and at times for informational messages. Common errors include pod scheduling failures, storage unavailability, missing secrets, missing storage classes. When the Aerospike cluster is not deployed as expected, events could help debug and find the cause.
To help troubleshoot issues, AKO generates events indicating state changes, errors and informational messages for the cluster it is working on.
Run kubectl get event --namespace [namespace] --field-selector involvedObject.name=[cluster name] to see the events generated by AKO for an Aerospike cluster. For example, this command displays the events generated for cluster aeroclutster in the aerospike namespace:
kubectl -n aerospike --field-selector involvedObject.name=aerocluster get eventsOutput:
Terminal window
To see all the events for the cluster’s namespace (aerospike in this case), run the following command
kubectl -n aerospike get eventsOutput:
Terminal window
Operator logs
AKOCTL collectinfo logs
akoctl is a Krew plugin for the Kubernetes Operator that uses the command collectinfo to collect logs from a cluster or namespace.
See AKOCTL for more information.
List the AKO pods:
kubectl -n operators get pod | grep aerospike-operator-controller-managerOutput:
Terminal window
Get the log for an AKO pod using the pod names above using the following command:
kubectl -n operators logs aerospike-operator-controller-manager-677f99497c-qrtcl -c managerAdd the -f flag to follow the logs continuously.
kubectl -n operators logs -f aerospike-operator-controller-manager-677f99497c-qrtcl -c managerThe series of steps AKO follows to apply user changes are logged along with errors and warnings.