Troubleshooting

An overview of a list of components to assist in troubleshooting.

Logging

To troubleshoot issues with the csi driver, you can look at logs from the secrets-store container of the csi driver pod running on the same node as your application pod:

kubectl get pod -o wide
# find the secrets store csi driver pod running on the same node as your application pod

kubectl logs csi-secrets-store-secrets-store-csi-driver-7x44t secrets-store

If the pod fails to start because of the inline volume mount, you can describe the pod to view mount failure errors and events:

kubectl describe pod <application pod>

It is always a good idea to include relevant logs from csi driver pod when opening a new issue.

Common Errors

SecretProviderClass not found

kubectl describe pod <application pod> shows:

  Warning  FailedMount  3s (x4 over 6s)  kubelet, kind-control-plane  MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to get secretproviderclass default/azure, error: secretproviderclasses.secrets-store.csi.x-k8s.io "azure" not found

The SecretProviderClass being referenced in the volumeMount needs to exist in the same namespace as the application pod.

Volume mount fails with secrets-store.csi.k8s.io not found in the list of registered CSI drivers

kubectl describe pod <application pod> shows:

  Warning  FailedMount  1s (x4 over 4s)  kubelet, kind-control-plane  MountVolume.SetUp failed for volume "secrets-store-inline" : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name secrets-store.csi.k8s.io not found in the list of registered CSI drivers

Secrets Store CSI Driver is deployed as a DaemonSet. The above error indicates the CSI driver pods aren’t running on the node.

  • If the node is tainted, then add a toleration for the taint in the Secrets Store CSI Driver DaemonSet.

  • Check to see if there are any node selectors preventing the Secrets Store CSI Driver pods from running on the node.

  • Check to see if the CSIDriver object has been deployed to the cluster:

    # This is the desired output. If the secrets-store.csi.k8s.io isn't found, then reinstall the driver.
    kubectl get csidriver
    NAME                       ATTACHREQUIRED   PODINFOONMOUNT   MODES       AGE
    secrets-store.csi.k8s.io   false            true             Ephemeral   110m
    

Mount fails with grpc: received message larger than max

If the files pulled in by the SecretProviderClass are larger than 4MiB you may observe FailedMount warnings with a message that includes grpc: received message larger than max. You can configure the driver to accept responses larger than 4MiB by specifying the --max-call-recv-msg-size=<size in bytes> argument to the secrets-store container in the csi-secrets-store DaemonSet.

Note that this may also increase memory resource consumption of the secrets-store container, so you should also consider increasing the memory limit as well.