Troubleshooting
An overview of a list of components to assist in troubleshooting.
Logging
To troubleshoot issues with the csi driver, you can look at logs from the secrets-store
container of the csi driver pod running on the same node as your application pod:
kubectl get pod -o wide
# find the secrets store csi driver pod running on the same node as your application pod
kubectl logs csi-secrets-store-secrets-store-csi-driver-7x44t secrets-store
If the pod fails to start because of the inline volume mount, you can describe the pod to view mount failure errors and events:
kubectl describe pod <application pod>
It is always a good idea to include relevant logs from csi driver pod when opening a new issue.
pprof
Starting the secrets-store
container in driver with --enable-pprof=true
will enable a debug http endpoint at --pprof-port
(default: 6065). Accessing this will also require port-forward
:
kubectl port-forward csi-secrets-store-secrets-store-csi-driver-7x44t secrets-store 6065:6065 &
curl localhost:6065/debug/pprof
Common Errors
SecretProviderClass
not found
kubectl describe pod <application pod>
shows:
Warning FailedMount 3s (x4 over 6s) kubelet, kind-control-plane MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to get secretproviderclass default/azure, error: secretproviderclasses.secrets-store.csi.x-k8s.io "azure" not found
The SecretProviderClass
being referenced in the volumeMount
needs to exist in the same namespace as the application pod.
Volume mount fails with secrets-store.csi.k8s.io
not found in the list of registered CSI drivers
kubectl describe pod <application pod>
shows:
Warning FailedMount 1s (x4 over 4s) kubelet, kind-control-plane MountVolume.SetUp failed for volume "secrets-store-inline" : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name secrets-store.csi.k8s.io not found in the list of registered CSI drivers
Secrets Store CSI Driver is deployed as a DaemonSet. The above error indicates the CSI driver pods aren’t running on the node.
-
If the node is tainted, then add a toleration for the taint in the Secrets Store CSI Driver DaemonSet.
-
Check to see if there are any node selectors preventing the Secrets Store CSI Driver pods from running on the node.
-
Check to see if the
CSIDriver
object has been deployed to the cluster:# This is the desired output. If the secrets-store.csi.k8s.io isn't found, then reinstall the driver. kubectl get csidriver NAME ATTACHREQUIRED PODINFOONMOUNT MODES AGE secrets-store.csi.k8s.io false true Ephemeral 110m
Mount fails with grpc: received message larger than max
If the files pulled in by the SecretProviderClass
are larger than 4MiB you may observe FailedMount
warnings with a
message that includes grpc: received message larger than max
. You can configure the driver to accept responses larger
than 4MiB by specifying the --max-call-recv-msg-size=<size in bytes>
argument to the secrets-store
container in the
csi-secrets-store
DaemonSet.
Note that this may also increase memory resource consumption of the secrets-store
container, so you should also
consider increasing the memory limit as well.