Kubernetes Secrets Store CSI Driver

Secrets Store CSI driver for Kubernetes secrets - Integrates secrets stores with Kubernetes via a Container Storage Interface (CSI) volume.

The Secrets Store CSI driver secrets-store.csi.k8s.io allows Kubernetes to mount multiple secrets, keys, and certs stored in enterprise-grade external secrets stores into their pods as a volume. Once the Volume is attached, the data in it is mounted into the container’s file system.

Want to help?

Join us to help define the direction and implementation of this project!

Features

  • Mounts secrets/keys/certs to pod using a CSI volume
  • Supports mounting multiple secrets store objects as a single volume
  • Supports multiple secrets stores as providers. Multiple providers can run in the same cluster simultaneously
  • Supports pod portability with the SecretProviderClass CRD
  • Supports windows containers (Kubernetes version v1.18+)
  • Supports sync with Kubernetes Secrets (Secrets Store CSI Driver v0.0.10+)
  • Support auto rotation of mounted contents and synced Kubernetes secret (Secrets Store CSI Driver v0.0.15+)

Supported Providers

Concepts

How it works

The diagram below illustrates how Secrets Store CSI volume works:

diagram

Similar to Kubernetes secrets, on pod start and restart, the Secrets Store CSI driver communicates with the provider using gRPC to retrieve the secret content from the external Secrets Store specified in the SecretProviderClass custom resource. Then the volume is mounted in the pod as tmpfs and the secret contents are written to the volume.

On pod delete, the corresponding volume is cleaned up and deleted.

Secrets Store CSI Driver

The Secrets Store CSI Driver is a daemonset that facilitates communication with every instance of Kubelet. Each driver pod has the following containers:

  • node-driver-registrar: Responsible for registering the CSI driver with Kubelet so that it knows which unix domain socket to issue the CSI calls on. This sidecar container is provider by the Kubernetes CSI team. See doc for more details.
  • secrets-store: Implements the CSI Node service gRPC services described in the CSI specification. It’s responsible for mount/unmount the volumes during pod creation/deletion. This component is developed and maintained in this repo.
  • liveness-probe: Responsible for monitoring the health of the CSI driver and reports to Kubernetes. This enables Kubernetes to automatically detect issues with the driver and restart the pod to try and fix the issue. This sidecar container is provider by the Kubernetes CSI team. See doc for more details.

Provider for the Secrets Store CSI Driver

The CSI driver communicates with the provider using gRPC to fetch the mount contents from external Secrets Store. Refer to doc for more details on the how to implement a provider for the driver and criteria for supported providers.

Currently supported providers:

Custom Resource Definitions (CRDs)

SecretProviderClass

The SecretProviderClass is a namespaced resource in Secrets Store CSI Driver that is used to provide driver configurations and provider-specific parameters to the CSI driver.

SecretProviderClass custom resource should have the following components:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: my-provider
spec:
  provider: vault                             # accepted provider options: azure or vault or gcp
  parameters:                                 # provider-specific parameters

Refer to the provider docs for required provider specific parameters.

Here is an example of a SecretProviderClass resource:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: my-provider
  namespace: default
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useManagedIdentity: "false"
    keyvaultName: "$KEYVAULT_NAME"
    objects: |
      array:
        - |
          objectName: $SECRET_NAME
          objectType: secret
          objectVersion: $SECRET_VERSION
        - |
          objectName: $KEY_NAME
          objectType: key
          objectVersion: $KEY_VERSION
    tenantId: "$TENANT_ID"

Reference the SecretProviderClass in the pod volumes when using the CSI driver:

volumes:
  - name: secrets-store-inline
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "my-provider"

NOTE: The SecretProviderClass needs to be created in the same namespace as the pod.

SecretProviderClassPodStatus

The SecretProviderClassPodStatus is a namespaced resource in Secrets Store CSI Driver that is created by the CSI driver to track the binding between a pod and SecretProviderClass. The SecretProviderClassPodStatus contains details about the current object versions that have been loaded in the pod mount.

The SecretProviderClassPodStatus is created by the CSI driver in the same namespace as the pod and SecretProviderClass with the name <pod name>-<namespace>-<secretproviderclass name>.

Here is an example of a SecretProviderClassPodStatus resource:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClassPodStatus
metadata:
  creationTimestamp: "2021-01-21T19:20:11Z"
  generation: 1
  labels:
    internal.secrets-store.csi.k8s.io/node-name: kind-control-plane
    manager: secrets-store-csi
    operation: Update
    time: "2021-01-21T19:20:11Z"
  name: nginx-secrets-store-inline-crd-dev-azure-spc
  namespace: dev
  ownerReferences:
  - apiVersion: v1
    kind: Pod
    name: nginx-secrets-store-inline-crd
    uid: 10f3e31c-d20b-4e46-921a-39e4cace6db2
  resourceVersion: "1638459"
  selfLink: /apis/secrets-store.csi.x-k8s.io/v1alpha1/namespaces/dev/secretproviderclasspodstatuses/nginx-secrets-store-inline-crd
  uid: 1d078ad7-c363-4147-a7e1-234d4b9e0d53
status:
  mounted: true
  objects:
  - id: secret/secret1
    version: c55925c29c6743dcb9bb4bf091be03b0
  - id: secret/secret2
    version: 7521273d0e6e427dbda34e033558027a
  podName: nginx-secrets-store-inline-crd
  secretProviderClassName: azure-spc
  targetPath: /var/lib/kubelet/pods/10f3e31c-d20b-4e46-921a-39e4cace6db2/volumes/kubernetes.io~csi/secrets-store-inline/mount

The pod for which the SecretProviderClassPodStatus was created is set as owner. When the pod is deleted, the SecretProviderClassPodStatus resources associated with the pod get automatically deleted.

Getting Started

This guide will walk you through the steps to configure and run the Secrets Store CSI driver on Kubernetes.

Installation

Install the Secrets Store CSI Driver and provider

Prerequisites

Supported kubernetes versions

Recommended Kubernetes version:

  • v1.16.0+ (For Linux)
  • v1.18.0+ (For Windows)

Deployment using Helm

Secrets Store CSI Driver allows users to customize their installation via Helm.

Recommended to use Helm3

helm repo add secrets-store-csi-driver https://raw.githubusercontent.com/kubernetes-sigs/secrets-store-csi-driver/master/charts
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --namespace kube-system

Running the above helm install command will install the Secrets Store CSI Driver on Linux nodes in the kube-system namespace.

Values

For a list of customizable values that can be injected when invoking helm install, please see the Helm chart configurations.

[Alternatively] Deployment using yamls

kubectl apply -f deploy/rbac-secretproviderclass.yaml
kubectl apply -f deploy/csidriver.yaml
kubectl apply -f deploy/secrets-store.csi.x-k8s.io_secretproviderclasses.yaml
kubectl apply -f deploy/secrets-store.csi.x-k8s.io_secretproviderclasspodstatuses.yaml
kubectl apply -f deploy/secrets-store-csi-driver.yaml

# If using the driver to sync secrets-store content as Kubernetes Secrets, deploy the additional RBAC permissions
# required to enable this feature
kubectl apply -f deploy/rbac-secretprovidersyncing.yaml

# If using the secret rotation feature, deploy the additional RBAC permissions
# required to enable this feature
kubectl apply -f deploy/rbac-secretproviderrotation.yaml

# [OPTIONAL] To deploy driver on windows nodes
kubectl apply -f deploy/secrets-store-csi-driver-windows.yaml

To validate the installer is running as expected, run the following commands:

kubectl get po --namespace=kube-system

You should see the Secrets Store CSI driver pods running on each agent node:

csi-secrets-store-qp9r8         3/3     Running   0          4m
csi-secrets-store-zrjt2         3/3     Running   0          4m

You should see the following CRDs deployed:

kubectl get crd
NAME                                               
secretproviderclasses.secrets-store.csi.x-k8s.io
secretproviderclasspodstatuses.secrets-store.csi.x-k8s.io

Use the Secrets Store CSI Driver with a Provider

Now that the Secrets Store CSI Driver has been deployed, select a provider from the supported provider list, then follow the installation steps for the provider:

Usage

Create your own SecretProviderClass Object

To use the Secrets Store CSI driver, create a SecretProviderClass custom resource to provide driver configurations and provider-specific parameters to the CSI driver.

A SecretProviderClass custom resource should have the following components:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: my-provider
spec:
  provider: vault                             # accepted provider options: azure or vault or gcp
  parameters:                                 # provider-specific parameters

Here is a sample SecretProviderClass custom resource

Update your Deployment Yaml

To ensure your application is using the Secrets Store CSI driver, update your deployment yaml to use the secrets-store.csi.k8s.io driver and reference the SecretProviderClass resource created in the previous step.

volumes:
  - name: secrets-store-inline
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "my-provider"

Here is a sample deployment yaml using the Secrets Store CSI driver.

Secret Content is Mounted on Pod Start

On pod start and restart, the driver will communicate with the provider using gRPC to retrieve the secret content from the external Secrets Store you have specified in the SecretProviderClass custom resource. Then the volume is mounted in the pod as tmpfs and the secret contents are written to the volume.

To validate, once the pod is started, you should see the new mounted content at the volume path specified in your deployment yaml.

kubectl exec secrets-store-inline -- ls /mnt/secrets-store/
foo

[OPTIONAL] Sync with Kubernetes Secrets

Refer to Sync as Kubernetes Secret for steps on syncing the secrets-store content as Kubernetes secret in addition to the mount.

[OPTIONAL] Set ENV VAR

Refer to Set as ENV var for steps on syncing the secrets-store content as Kubernetes secret and using the secret for env variables in the deployment.

[OPTIONAL] Enable Auto Rotation of Secrets

You can setup the Secrets Store CSI Driver to periodically update the pod mount and Kubernetes Secret with the latest content from external secrets-store. Refer to Secret Auto Rotation for steps on enabling auto rotation.

Upgrades

This page includes instructions for upgrading the driver to the latest version.

helm upgrade csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --namespace=NAMESPACE

Set NAMESPACE to the same namespace where the driver was originally installed, (i.e. kube-system)

If you are upgrading from one of the following versions there may be additional steps that you should take.

pre v0.0.23

v0.0.23 sets syncSecret.enabled=false by default. This means the RBAC clusterrole and clusterrolebinding required for sync mounted content as Kubernetes secret will no longer be created by default as part of helm install/upgrade. If you’re using the driver to sync mounted content as Kubernetes secret, you’ll need to set syncSecret.enabled=true as part of helm install/upgrade.

pre v0.0.20

v0.0.20 removed support for non-gRPC based providers. Follow your provider documentation to upgrade providers to use gRPC before upgrading the driver to v0.0.20 or greater.

pre v0.0.18

v0.0.17 and earlier installed the driver to the default namespace when using the YAML based install. Newer versions of the driver YAML files install the driver to the kube-system namespace. After applying the new YAML files to your cluster run the following to clean up old resources:

kubectl delete daemonset csi-secrets-store --namespace=default
kubectl delete daemonset csi-secrets-store-windows --namespace=default
kubectl delete serviceaccount secrets-store-csi-driver --namespace=default

pre v0.0.12

The SecretProviderClass needs to be in the same namespace as the pod referencing it as of v0.0.12.

Defining driver configuration and provider-specific parameters to the CSI driver in pod.Spec[].Volumes has been deprecated in v0.0.12. It is now mandatory to use SecretProviderClass custom resource.

Topics

This section contains information about various features supported by Secrets Store CSI Driver.

Metrics provided by Secrets Store CSI Driver

The Secrets Store CSI Driver uses opentelemetry for reporting metrics. This project is under active development

Prometheus is the only exporter that’s currently supported with the driver.

List of metrics provided by the driver

MetricDescriptionTags
total_node_publishTotal number of successful volume mount requestsos_type=<runtime os>
provider=<provider name>
total_node_unpublishTotal number of successful volume unmount requestsos_type=<runtime os>
total_node_publish_errorTotal number of errors with volume mount requestsos_type=<runtime os>
provider=<provider name>
error_type=<error code>
total_node_unpublish_errorTotal number of errors with volume unmount requestsos_type=<runtime os>
total_sync_k8s_secretTotal number of k8s secrets syncedos_type=<runtime os>
provider=<provider name>
sync_k8s_secret_duration_secDistribution of how long it took to sync k8s secretos_type=<runtime os>
total_rotation_reconcileTotal number of rotation reconcilesos_type=<runtime os>
rotated=<true or false>
total_rotation_reconcile_errorTotal number of rotation reconciles with erroros_type=<runtime os>
rotated=<true or false>
error_type=<error code>
rotation_reconcile_duration_secDistribution of how long it took to rotate secrets-store content for podsos_type=<runtime os>

Sample Metrics output

# HELP sync_k8s_secret_duration_sec Distribution of how long it took to sync k8s secret
# TYPE sync_k8s_secret_duration_sec histogram
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="0.1"} 0
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="0.2"} 0
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="0.3"} 0
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="0.4"} 1
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="0.5"} 1
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="1"} 1
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="1.5"} 1
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="2"} 1
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="2.5"} 1
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="3"} 1
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="5"} 1
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="10"} 1
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="15"} 1
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="30"} 1
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="+Inf"} 1
sync_k8s_secret_duration_sec_sum{os_type="linux"} 0.3115892
sync_k8s_secret_duration_sec_count{os_type="linux"} 1
# HELP total_node_publish Total number of node publish calls
# TYPE total_node_publish counter
total_node_publish{os_type="linux",provider="azure"} 1
# HELP total_node_publish_error Total number of node publish calls with error
# TYPE total_node_publish_error counter
total_node_publish_error{error_type="ProviderBinaryNotFound",os_type="linux",provider="azure"} 2
total_node_publish_error{error_type="SecretProviderClassNotFound",os_type="linux",provider=""} 4
# HELP total_node_unpublish Total number of node unpublish calls
# TYPE total_node_unpublish counter
total_node_unpublish{os_type="linux"} 1
# HELP total_sync_k8s_secret Total number of k8s secrets synced
# TYPE total_sync_k8s_secret counter
total_sync_k8s_secret{os_type="linux",provider="azure"} 1

Auto rotation of mounted contents and synced Kubernetes Secrets

  • Design doc: Rotation Design
  • Feature State: Secrets Store CSI Driver v0.0.15 [alpha]

When the secret/key is updated in external secrets store after the initial pod deployment, the updated secret will be periodically updated in the pod mount and the Kubernetes Secret.

Depending on how the application consumes the secret data:

  1. Mount Kubernetes secret as a volume: Use auto rotation feature + Sync K8s secrets feature in Secrets Store CSI Driver, application will need to watch for changes from the mounted Kubernetes Secret volume. When the Kubernetes Secret is updated by the CSI Driver, the corresponding volume contents are automatically updated.
  2. Application reads the data from container’s filesystem: Use rotation feature in Secrets Store CSI Driver, application will need to watch for the file change from the volume mounted by the CSI driver.
  3. Using Kubernetes secret for environment variable: The pod needs to be restarted to get the latest secret as environment variable.
    1. Use something like https://github.com/stakater/Reloader to watch for changes on the synced Kubernetes secret and do rolling upgrades on pods

Enable auto rotation

NOTE: This alpha feature is not enabled by default.

To enable auto rotation, enable the --enable-secret-rotation feature gate for the secrets-store container in the Secrets Store CSI Driver pods. The rotation poll interval can be configured using --rotation-poll-interval. The default rotation poll interval is 2m. If using helm to install the driver, set enableSecretRotation: true and configure the rotation poll interval by setting rotationPollInterval. The rotation poll interval can be tuned based on how frequently the mounted contents for all pods and Kubernetes secrets need to be resynced to the latest.

  • The Secrets Store CSI Driver will update the pod mount and the Kubernetes Secret defined in secretObjects of SecretProviderClass periodically based on the rotation poll interval to the latest value.
  • If the SecretProviderClass is updated after the pod was initially created
    • Adding/deleting objects and updating keys in existing secretObjects - the pod mount and Kubernetes secret will be updated with the new objects added to the SecretProviderClass.
    • Adding new secretObject to the existing secretObjects - the Kubernetes secret will be created by the controller.

How to view the current secret versions loaded in pod mount

The Secrets Store CSI Driver creates a custom resource SecretProviderClassPodStatus to track the binding between a pod and SecretProviderClass. This SecretProviderClassPodStatus status also contains the details about the secrets and versions currently loaded in the pod mount.

The SecretProviderClassPodStatus is created in the same namespace as the pod with the name <pod name>-<namespace>-<secretproviderclass name>

➜ kubectl get secretproviderclasspodstatus nginx-secrets-store-inline-crd-default-azure-spc -o yaml
...
status:
  mounted: true
  objects:
  - id: secret/secret1
    version: b82206cb5ac249918008b0b97fd1fd66
  - id: key/key1
    version: 7cc095105411491b84fe1b92ebbcf01a
  podName: nginx-secrets-store-inline-multiple-crd
  secretProviderClassName: azure-spc
  targetPath: /var/lib/kubelet/pods/1b7b0740-62d5-4776-a0df-90d060ef35ba/volumes/kubernetes.io~csi/secrets-store-inline-0/mount

Limitations

The auto rotation feature is only supported with providers that have implemented gRPC server for enabling driver-provider communication.

Sync as Kubernetes Secret

Examples
  • SecretProviderClass
apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: azure-sync
spec:
  provider: azure
  secretObjects:                                 # [OPTIONAL] SecretObject defines the desired state of synced K8s secret objects
  - secretName: foosecret
    type: Opaque
    labels:                                   
      environment: "test"
    data: 
    - objectName: secretalias                    # name of the mounted content to sync. this could be the object name or object alias 
      key: username
  parameters:
    usePodIdentity: "true"                      
    keyvaultName: "$KEYVAULT_NAME"               # the name of the KeyVault
    objects: |
      array:
        - |
          objectName: $SECRET_NAME
          objectType: secret                     # object types: secret, key or cert
          objectAlias: secretalias
          objectVersion: $SECRET_VERSION         # [OPTIONAL] object versions, default to latest if empty
        - |
          objectName: $KEY_NAME
          objectType: key
          objectVersion: $KEY_VERSION
    tenantId: "tid"                             # the tenant ID of the KeyVault
  • Pod yaml
kind: Pod
apiVersion: v1
metadata:
  name: secrets-store-inline
spec:
  containers:
    - name: busybox
      image: k8s.gcr.io/e2e-test-images/busybox:1.29
      command:
      - "/bin/sleep"
      - "10000"
      volumeMounts:
      - name: secrets-store01-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
  volumes:
    - name: secrets-store01-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "azure-sync"

In some cases, you may want to create a Kubernetes Secret to mirror the mounted content. Use the optional secretObjects field to define the desired state of the synced Kubernetes secret objects. The volume mount is required for the Sync With Kubernetes Secrets

NOTE: If the provider supports object alias for the mounted file, then make sure the objectName in secretObjects matches the name of the mounted content. This could be the object name or the object alias.

The secrets will only sync once you start a pod mounting the secrets. Solely relying on the syncing with Kubernetes secrets feature thus does not work. When all the pods consuming the secret are deleted, the Kubernetes secret is also deleted.

A SecretProviderClass custom resource should have the following components:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: my-provider
spec:
  provider: vault                             # accepted provider options: azure or vault or gcp
  secretObjects:                              # [OPTIONAL] SecretObject defines the desired state of synced K8s secret objects
  - data:
    - key: username                           # data field to populate
      objectName: foo1                        # name of the mounted content to sync. this could be the object name or the object alias
    secretName: foosecret                     # name of the Kubernetes Secret object
    type: Opaque                              # type of the Kubernetes Secret object e.g. Opaque, kubernetes.io/tls

NOTE: Here is the list of supported Kubernetes Secret types: Opaque, kubernetes.io/basic-auth, bootstrap.kubernetes.io/token, kubernetes.io/dockerconfigjson, kubernetes.io/dockercfg, kubernetes.io/ssh-auth, kubernetes.io/service-account-token, kubernetes.io/tls.

Set as ENV var

Examples
  • SecretProviderClass
apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: azure-sync
spec:
  provider: azure
  secretObjects:                                 # [OPTIONAL] SecretObject defines the desired state of synced K8s secret objects
  - secretName: foosecret
    type: Opaque
    labels:                                   
      environment: "test"
    data: 
    - objectName: secretalias                    # name of the mounted content to sync. this could be the object name or object alias 
      key: username
  parameters:
    usePodIdentity: "false"                      
    keyvaultName: "$KEYVAULT_NAME"               # the name of the KeyVault
    objects: |
      array:
        - |
          objectName: $SECRET_NAME
          objectType: secret                     # object types: secret, key or cert
          objectAlias: secretalias
          objectVersion: $SECRET_VERSION         # [OPTIONAL] object versions, default to latest if empty
        - |
          objectName: $KEY_NAME
          objectType: key
          objectVersion: $KEY_VERSION
    tenantId: "tid"                             # the tenant ID of the KeyVault
  • Pod yaml
kind: Pod
apiVersion: v1
metadata:
  name: secrets-store-inline
spec:
  containers:
  - name: busybox
    image: k8s.gcr.io/e2e-test-images/busybox:1.29
    command:
    - "/bin/sleep"
    - "10000"
    volumeMounts:
    - name: secrets-store01-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
    env:
    - name: SECRET_USERNAME
      valueFrom:
        secretKeyRef:
          name: foosecret
          key: username
  volumes:
  - name: secrets-store01-inline
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "azure-sync"

Once the secret is created, you may wish to set an ENV VAR in your deployment to reference the new Kubernetes secret.

spec:
  containers:
  - image: k8s.gcr.io/e2e-test-images/busybox:1.29
    name: busybox
    command:
    - "/bin/sleep"
    - "10000"
    env:
    - name: SECRET_USERNAME
      valueFrom:
        secretKeyRef:
          name: foosecret
          key: username

Best Practices

  1. Deploy the driver and providers into the kube-system or a separate dedicated namespace.

    The driver is installed as a DaemonSet with the ability mount kubelet hostPath volumes and view pod service account tokens. It should be treated as privileged and regular cluster users should not have permissions to deploy or modify the driver.

  2. Do not grant regular cluster users permissions to modify SecretProviderClassPodStatus resources.

    The SecretProviderClassPodStatus CRD is used by the driver to keep track of mounted resources. Manually editing this resource could have unexpected consequences to the system health and in particular modifying SecretProviderClassPodStatus/status may have security implications.

  3. Disable Secret sync if not needed.

    If you do not intend to use the Secret syncing feature, do not install the RBAC permissions that allow the driver to access cluster Secret objects.

    This can be done by setting syncSecret.enabled = false when installing with helm.

  4. Enable KMS application wrapping if using Secret sync.

    If you need to synchronise your external secrets to Kubernetes Secrets consider configuring encryption of data at rest

    This will ensure that data is encrypted before it is stored in etcd.

  5. Keep the driver up to date.

    Subscribe to the kubernetes-secrets-store-csi-driver mailing list to be notified of new releases and security announcements.

    Consider using the Github Watch feature to subscribe to releases as well.

    Always be sure to review the release notes before upgrading.

  6. When evaluating this driver consider the following threats:

    • When a secret is accessible on the filesystem, application vulnerabilities like directory traversal attacks can become higher severity as the attacker may gain the ability read the secret material.
    • When a secret is consumed through environment variables, misconfigurations such as enabling a debug endpoints or including dependencies that log process environment details may leak secrets.
    • When syncing secret material to Kubernetes Secrets, consider whether the access controls on that data store are sufficiently narrow in scope.

    If possible, directly integrating with a purpose built secrets API may offer the best security tradeoffs.

Providers

This project features a pluggable provider interface developers can implement that defines the actions of the Secrets Store CSI driver. This enables retrieval of sensitive objects stored in an enterprise-grade external secrets store into Kubernetes while continue to manage these objects outside of Kubernetes.

Criteria for Supported Providers

Here is a list of criteria for supported provider:

  1. Code audit of the provider implementation to ensure it adheres to the required provider-driver interface - Implementing a Provider for Secrets Store CSI Driver
  2. Add provider to the e2e test suite to demonstrate it functions as expected. Please use existing providers e2e tests as a reference.
  3. If any update is made by a provider (not limited to security updates), the provider is expected to update the provider’s e2e test in this repo.

Removal from Supported Providers

Failure to adhere to the Criteria for Supported Providers will result in the removal of the provider from the supported list and subject to another review before it can be added back to the list of supported providers.

When a provider’s e2e tests are consistently failing with the latest version of the driver, the driver maintainers will coordinate with the provider maintainers to provide a fix. If the test failures are not resolved within 4 weeks, then the provider will be removed from the list of supported providers.

Implementing a Provider for Secrets Store CSI Driver

This document highlights the implementation steps for adding a secrets-store-csi-driver provider.

Implementation details

The driver as of v0.0.14 adds an option to use gRPC to communicate with the provider. This is an alpha feature and is introduced with a feature flag --grpc-supported-providers. The --grpc-supported-providers is a ; delimited list of all providers that support gRPC for communication. This flag will not be necessary after v0.0.21 since this is the only supported communication mechanism.

Example usage: --grpc-supported-providers=provider1;provider2

To implement a secrets-store-csi-driver provider, you can develop a new provider gRPC server using the stub file available for Go.

  • Use the functions and data structures in the stub file: service.pb.go to develop the server code
    • The stub file and proto file are shared and hosted in the driver. Vendor-in the stub file and proto file in the provider
    • fake server example
  • Provider runs as a daemonset and is deployed on the same host(s) as the secrets-store-csi-driver pods
  • Provider Unix Domain Socket volume path. The default volume path for providers is /etc/kubernetes/secrets-store-csi-providers. Add the Unix Domain Socket to the dir in the format /etc/kubernetes/secrets-store-csi-providers/<provider name>.sock
  • The <provider name> in <provider name>.sock must match the regular expression ^[a-zA-Z0-9_-]{0,30}$
  • Provider mounts <kubelet root dir>/pods (default: /var/lib/kubelet/pods) with HostToContainer mount propagation to be able to write the external secrets store content to the volume target path

See design doc for more details.

Troubleshooting

An overview of a list of components to assist in troubleshooting.

Logging

To troubleshoot issues with the csi driver, you can look at logs from the secrets-store container of the csi driver pod running on the same node as your application pod:

kubectl get pod -o wide
# find the secrets store csi driver pod running on the same node as your application pod

kubectl logs csi-secrets-store-secrets-store-csi-driver-7x44t secrets-store

If the pod fails to start because of the inline volume mount, you can describe the pod to view mount failure errors and events:

kubectl describe pod <application pod>

It is always a good idea to include relevant logs from csi driver pod when opening a new issue.

Common Errors

SecretProviderClass not found

kubectl describe pod <application pod> shows:

  Warning  FailedMount  3s (x4 over 6s)  kubelet, kind-control-plane  MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to get secretproviderclass default/azure, error: secretproviderclasses.secrets-store.csi.x-k8s.io "azure" not found

The SecretProviderClass being referenced in the volumeMount needs to exist in the same namespace as the application pod.

Volume mount fails with secrets-store.csi.k8s.io not found in the list of registered CSI drivers

kubectl describe pod <application pod> shows:

  Warning  FailedMount  1s (x4 over 4s)  kubelet, kind-control-plane  MountVolume.SetUp failed for volume "secrets-store-inline" : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name secrets-store.csi.k8s.io not found in the list of registered CSI drivers

Secrets Store CSI Driver is deployed as a DaemonSet. The above error indicates the CSI driver pods aren’t running on the node.

  • If the node is tainted, then add a toleration for the taint in the Secrets Store CSI Driver DaemonSet.

  • Check to see if there are any node selectors preventing the Secrets Store CSI Driver pods from running on the node.

  • Check to see if the CSIDriver object has been deployed to the cluster:

    # This is the desired output. If the secrets-store.csi.k8s.io isn't found, then reinstall the driver.
    kubectl get csidriver
    NAME                       ATTACHREQUIRED   PODINFOONMOUNT   MODES       AGE
    secrets-store.csi.k8s.io   false            true             Ephemeral   110m
    

Mount fails with grpc: received message larger than max

If the files pulled in by the SecretProviderClass are larger than 4MiB you may observe FailedMount warnings with a message that includes grpc: received message larger than max. You can configure the driver to accept responses larger than 4MiB by specifying the --max-call-recv-msg-size=<size in bytes> argument to the secrets-store container in the csi-secrets-store DaemonSet.

Note that this may also increase memory resource consumption of the secrets-store container, so you should also consider increasing the memory limit as well.

Load tests

This document highlights the results from load tests using secrets-store-csi-driver v0.0.21.

Note: Refer to doc for more details on the optimization done as part of v0.0.21 release.

The results posted here can be used as a guide for configuring resource and memory limits for the CSI driver daemonset pods.

Testing Environment

  1. 250 nodes Azure Kubernetes Service cluster
  2. 3500 Kubernetes secrets in the cluster
    • These secrets were pre-configured to ensure existing Kubernetes secrets doesn’t impact the memory for the CSI driver.
  3. 7250 pods running in the cluster
    • These pods were pre-configured to ensure existing Kubernetes pods doesn’t impact the memory for the CSI driver.

The Secrets Store CSI Driver and Azure Keyvault Provider were deployed to the cluster.

Secrets Store CSI Driver features enabled:

  1. Sync as Kubernetes secret
  2. Secret Auto rotation
    • Rotation Poll Interval: 2m

Testing scenarios

10000 pods with CSI volume

  1. 10000 pods with CSI volume.
    • Total number of pods in the cluster = 7250 + 10000 = 17250 pods.
  2. SecretProviderClass with syncing 2 Kubernetes secrets.
➜ kubectl top pods -l app=csi-secrets-store -n kube-system --sort-by=memory
NAME                      CPU(cores)   MEMORY(bytes)
csi-secrets-store-kd2bc   3m           54Mi
csi-secrets-store-wx6z9   3m           52Mi
csi-secrets-store-6gjqq   3m           52Mi
csi-secrets-store-knl5g   4m           52Mi
csi-secrets-store-9lzzn   4m           51Mi

The current default memory and resource limits have been configured based on the above tests.

Understanding Secrets Store CSI Driver memory consumption

As of Secrets Store CSI Driver v0.0.21, the memory consumption for the driver is dependent on:

  1. Number of pods on the same node as the driver pod.
  2. Number of secrets with a. secrets-store.csi.k8s.io/managed=true label. This label is set for all the secrets created by the Secrets Store CSI Driver. b. secrets-store.csi.k8s.io/used=true label. This label needs to be set for all nodePublishSecretRef.
  3. Number of SecretProviderClass across all namespaces.
  4. Number of SecretProviderClassPodStatus created by Secrets Store CSI Driver for the pod on the same node as the application pod.
    1. Secrets Store CSI Driver creates a SecretProviderClassPodStatus to map pod to SecretProviderClass. See doc for more details.

Testing

Unit Tests

Run unit tests locally with make test.

End-to-end Tests

End-to-end tests automatically runs on Prow when a PR is submitted. If you want to run using a local or remote Kubernetes cluster, make sure to have kubectl, helm and bats set up in your local environment and then run make e2e-azure, make e2e-vault or make e2e-gcp with custom images.

Job config for test jobs run for each PR in prow can be found here

Known Limitations

This document highlights the current limitations when using secrets-store-csi-driver.

Mounted content and Kubernetes Secret not updated

  • When the secret/key is updated in external secrets store after the initial pod deployment, the updated secret is not automatically reflected in the pod mount or the Kubernetes secret.
  • When the SecretProviderClass is updated after the pod was initially created.
  • Adding/deleting objects and updating keys in existing secretObjects doesn’t result in update of Kubernetes secrets.

The CSI driver is invoked by kubelet only during the pod volume mount. So subsequent changes in the SecretProviderClass after the pod has started doesn’t trigger an update to the content in volume mount or Kubernetes secret.

Enable Secret autorotation feature has been released in v0.0.15+. Refer to doc and design doc for more details.

How to fetch the latest content with release v0.0.14 and earlier or without Auto rotation feature enabled?

  1. If the SecretProviderClass has secretObjects defined, then delete the Kubernetes secret.
  2. Restart the application pod.

When the pod is recreated, kubelet invokes the CSI driver for mounting the volume. As part of this mount request, the latest content will be fetched from external secrets store and populated in the pod. The same content is then mirrored in the Kubernetes secret data.