Kubernetes Secrets Store CSI Driver

Secrets Store CSI Driver for Kubernetes secrets - Integrates secrets stores with Kubernetes via a Container Storage Interface (CSI) volume.

The Secrets Store CSI Driver secrets-store.csi.k8s.io allows Kubernetes to mount multiple secrets, keys, and certs stored in enterprise-grade external secrets stores into their pods as a volume. Once the Volume is attached, the data in it is mounted into the container’s file system.

Want to help?

Join us to help define the direction and implementation of this project!

Join the #csi-secrets-store channel on Kubernetes Slack.
Join the Mailing list to receive notifications for releases, security announcements, etc.
Use GitHub Issues to file bugs, request features, or ask questions asynchronously.
Join biweekly community meetings to discuss development, issues, use cases, etc.

Project Status

Driver	Compatible Kubernetes	`secrets-store.csi.x-k8s.io` Versions
v1.4.0	1.19+	`v1`, `v1alpha1 [DEPRECATED]`
v1.3.4	1.19+	`v1`, `v1alpha1 [DEPRECATED]`

See Release Management for additional details on versioning. We aim to release a new minor version every month and intend to support the latest 2 minor versions of the driver.

Features

Driver Core Functionality (Stable)

Multiple external secrets store providers
Pod portability with the SecretProviderClass CustomResourceDefinition
Mounts secrets/keys/certs to pod using a CSI Inline volume
Mount multiple secrets store objects as a single volume
Linux and Windows containers

Alpha Functionality

These features are not stable. If you use these be sure to consult the upgrade instructions with each upgrade.

Auto rotation of mounted contents and synced Kubernetes secret
Sync with Kubernetes Secrets

Supported Providers

Concepts

How it works

The diagram below illustrates how Secrets Store CSI volume works:

diagram

Similar to Kubernetes secrets, on pod start and restart, the Secrets Store CSI driver communicates with the provider using gRPC to retrieve the secret content from the external Secrets Store specified in the SecretProviderClass custom resource. Then the volume is mounted in the pod as tmpfs and the secret contents are written to the volume.

On pod delete, the corresponding volume is cleaned up and deleted.

Secrets Store CSI Driver

The Secrets Store CSI Driver is a daemonset that facilitates communication with every instance of Kubelet. Each driver pod has the following containers:

node-driver-registrar: Responsible for registering the CSI driver with Kubelet so that it knows which unix domain socket to issue the CSI calls on. This sidecar container is provider by the Kubernetes CSI team. See doc for more details.
secrets-store: Implements the CSI Node service gRPC services described in the CSI specification. It’s responsible for mount/unmount the volumes during pod creation/deletion. This component is developed and maintained in this repo.
liveness-probe: Responsible for monitoring the health of the CSI driver and reports to Kubernetes. This enables Kubernetes to automatically detect issues with the driver and restart the pod to try and fix the issue. This sidecar container is provider by the Kubernetes CSI team. See doc for more details.

Provider for the Secrets Store CSI Driver

The CSI driver communicates with the provider using gRPC to fetch the mount contents from external Secrets Store. Refer to doc for more details on the how to implement a provider for the driver and criteria for supported providers.

Currently supported providers:

Security

The Secrets Store CSI Driver daemonset runs as root in a privileged pod. This is because the daemonset is responsible for creating new tmpfs filesystems and mounting them into existing pod filesystems within the node’s hostPath. root is necessary for the mount syscall and other filesystem operations and privileged is required for to use mountPropagation: Bidirectional to modify other running pod’s filesystems.

The provider plugins are also required to run as root (though privileged should not be necessary). This is because the provider plugin must create a unix domain socket in a hostPath for the driver to connect to.

Further, service account tokens for pods that require secrets may be forwarded from the kubelet process to the driver and then to provider plugins. This allows the provider to impersonate the pod when contacting the external secret API.

Note: On Windows hosts secrets will be written to the node’s filesystem which may be persistent storage. This contrasts with Linux where a tmpfs is used to try to ensure that secret material is never persisted.

Note: Kubernetes 1.22 introduced a way to configure nodes to use swap memory, however if this is used then secret material may be persisted to the node’s disk. To ensure that secrets are not written to persistent disk ensure failSwapOn is set to true (which is the default).

Security implications of using Secrets Store CSI driver

Anyone who has access to the namespace or its resources can exec and view the secrets. However, this behavior is expected, as entities with such access are expected to perform these operations. One potential solution is to disable exec if a more stringent security posture is required. Alternatively, using distroless images for applications can mitigate this, as exec won’t work. A similar argument can be made for individuals with access to the underlying infrastructure or nodes, who can access the secrets by SSHing into the node. However, typical end users do not have this level of access. If an end user does gain such access, it indicates a compromised infrastructure/cluster, and the recommended solution is to restrict access to the cluster/infrastructure.

Encrypting mounted content can be the solution to further protect the secrets. However, this introduces additional operational overhead, such as managing encryption keys and addressing key rotation. Key management becomes a crucial aspect similar to secrets management.

When we look from the perspective of an application and what access does it have, for instance, the Ingress Controller application which requires cluster-wide access to Kubernetes Secrets. If a component like Ingress is compromised, it could jeopardize all secrets in the cluster. This is where the Secrets Store CSI driver proves valuable, as it can mount/sync only the necessary TLS certificates on the Ingress Pod, reducing the blast radius.

Custom Resource Definitions (CRDs)

SecretProviderClass

The SecretProviderClass is a namespaced resource in Secrets Store CSI Driver that is used to provide driver configurations and provider-specific parameters to the CSI driver.

SecretProviderClass custom resource should have the following components:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: my-provider
spec:
  provider: vault                             # accepted provider options: akeyless or azure or vault or gcp
  parameters:                                 # provider-specific parameters

Refer to the provider docs for required provider specific parameters.

Here is an example of a SecretProviderClass resource:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: my-provider
  namespace: default
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useManagedIdentity: "false"
    keyvaultName: "$KEYVAULT_NAME"
    objects: |
      array:
        - |
          objectName: $SECRET_NAME
          objectType: secret
          objectVersion: $SECRET_VERSION
        - |
          objectName: $KEY_NAME
          objectType: key
          objectVersion: $KEY_VERSION
    tenantId: "$TENANT_ID"

Reference the SecretProviderClass in the pod volumes when using the CSI driver:

volumes:
  - name: secrets-store-inline
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "my-provider"

NOTE: The SecretProviderClass needs to be created in the same namespace as the pod.

SecretProviderClassPodStatus

The SecretProviderClassPodStatus is a namespaced resource in Secrets Store CSI Driver that is created by the CSI driver to track the binding between a pod and SecretProviderClass. The SecretProviderClassPodStatus contains details about the current object versions that have been loaded in the pod mount.

The SecretProviderClassPodStatus is created by the CSI driver in the same namespace as the pod and SecretProviderClass with the name <pod name>-<namespace>-<secretproviderclass name>.

Here is an example of a SecretProviderClassPodStatus resource:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClassPodStatus
metadata:
  creationTimestamp: "2021-01-21T19:20:11Z"
  generation: 1
  labels:
    internal.secrets-store.csi.k8s.io/node-name: kind-control-plane
    manager: secrets-store-csi
    operation: Update
    time: "2021-01-21T19:20:11Z"
  name: nginx-secrets-store-inline-crd-dev-azure-spc
  namespace: dev
  ownerReferences:
  - apiVersion: v1
    kind: Pod
    name: nginx-secrets-store-inline-crd
    uid: 10f3e31c-d20b-4e46-921a-39e4cace6db2
  resourceVersion: "1638459"
  selfLink: /apis/secrets-store.csi.x-k8s.io/v1/namespaces/dev/secretproviderclasspodstatuses/nginx-secrets-store-inline-crd
  uid: 1d078ad7-c363-4147-a7e1-234d4b9e0d53
status:
  mounted: true
  objects:
  - id: secret/secret1
    version: c55925c29c6743dcb9bb4bf091be03b0
  - id: secret/secret2
    version: 7521273d0e6e427dbda34e033558027a
  podName: nginx-secrets-store-inline-crd
  secretProviderClassName: azure-spc
  targetPath: /var/lib/kubelet/pods/10f3e31c-d20b-4e46-921a-39e4cace6db2/volumes/kubernetes.io~csi/secrets-store-inline/mount

The pod for which the SecretProviderClassPodStatus was created is set as owner. When the pod is deleted, the SecretProviderClassPodStatus resources associated with the pod get automatically deleted.

Getting Started

This guide will walk you through the steps to configure and run the Secrets Store CSI driver on Kubernetes.

Installation

Install the Secrets Store CSI Driver

Prerequisites

Supported kubernetes versions

Secrets Store CSI Driver will maintain support for all actively supported Kubernetes minor releases per Kubernetes Supported Versions policy. Check out the Kubernetes releases page for the latest supported Kubernetes releases.

Deployment using Helm

Secrets Store CSI Driver allows users to customize their installation via Helm.

Recommended to use Helm3

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --namespace kube-system

Running the above helm install command will install the Secrets Store CSI Driver on Linux nodes in the kube-system namespace.

Optional Values

Notably the following feature must be explicitly enabled:

Feature	Helm Parameter
Sync as Kubernetes secret	`syncSecret.enabled=true`
Secret Auto rotation	`enableSecretRotation=true`

For a list of customizable values that can be injected when invoking helm install, please see the Helm chart configurations.

[Alternatively] Deployment using yamls

kubectl apply -f deploy/rbac-secretproviderclass.yaml
kubectl apply -f deploy/csidriver.yaml
kubectl apply -f deploy/secrets-store.csi.x-k8s.io_secretproviderclasses.yaml
kubectl apply -f deploy/secrets-store.csi.x-k8s.io_secretproviderclasspodstatuses.yaml
kubectl apply -f deploy/secrets-store-csi-driver.yaml

# If using the driver to sync secrets-store content as Kubernetes Secrets, deploy the additional RBAC permissions
# required to enable this feature
kubectl apply -f deploy/rbac-secretprovidersyncing.yaml

# If using the secret rotation feature, deploy the additional RBAC permissions
# required to enable this feature
kubectl apply -f deploy/rbac-secretproviderrotation.yaml

# If using the CSI Driver token requests feature (https://kubernetes-csi.github.io/docs/token-requests.html) to use
# pod/workload identity to request a token and use with providers
kubectl apply -f deploy/rbac-secretprovidertokenrequest.yaml

# [OPTIONAL] To deploy driver on windows nodes
kubectl apply -f deploy/secrets-store-csi-driver-windows.yaml

To validate the installer is running as expected, run the following commands:

kubectl get po --namespace=kube-system

You should see the Secrets Store CSI driver pods running on each agent node:

csi-secrets-store-qp9r8         3/3     Running   0          4m
csi-secrets-store-zrjt2         3/3     Running   0          4m

You should see the following CRDs deployed:

kubectl get crd
NAME                                               
secretproviderclasses.secrets-store.csi.x-k8s.io
secretproviderclasspodstatuses.secrets-store.csi.x-k8s.io

Install External Secret Providers

Now that the Secrets Store CSI Driver has been deployed, select a provider from the supported provider list, then follow the installation steps for the provider:

Usage

Create your own SecretProviderClass Object

To use the Secrets Store CSI driver, create a SecretProviderClass custom resource to provide driver configurations and provider-specific parameters to the CSI driver.

A SecretProviderClass custom resource should have the following components:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: my-provider
spec:
  provider: vault                             # accepted provider options: akeyless or azure or vault or gcp
  parameters:                                 # provider-specific parameters

Here is a sample SecretProviderClass custom resource

Update your Deployment Yaml

To ensure your application is using the Secrets Store CSI driver, update your deployment yaml to use the secrets-store.csi.k8s.io driver and reference the SecretProviderClass resource created in the previous step.

volumes:
  - name: secrets-store-inline
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "my-provider"

Here is a sample deployment yaml using the Secrets Store CSI driver.

Secret Content is Mounted on Pod Start

On pod start and restart, the driver will communicate with the provider using gRPC to retrieve the secret content from the external Secrets Store you have specified in the SecretProviderClass custom resource. Then the volume is mounted in the pod as tmpfs and the secret contents are written to the volume.

To validate, once the pod is started, you should see the new mounted content at the volume path specified in your deployment yaml.

kubectl exec secrets-store-inline -- ls /mnt/secrets-store/
foo

[OPTIONAL] Sync with Kubernetes Secrets

Refer to Sync as Kubernetes Secret for steps on syncing the secrets-store content as Kubernetes secret in addition to the mount.

[OPTIONAL] Set ENV VAR

Refer to Set as ENV var for steps on syncing the secrets-store content as Kubernetes secret and using the secret for env variables in the deployment.

[OPTIONAL] Enable Auto Rotation of Secrets

You can setup the Secrets Store CSI Driver to periodically update the pod mount and Kubernetes Secret with the latest content from external secrets-store. Refer to Secret Auto Rotation for steps on enabling auto rotation.

Upgrades

This page includes instructions for upgrading the driver to the latest version.

helm upgrade csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --namespace=NAMESPACE

Set NAMESPACE to the same namespace where the driver was originally installed, (i.e. kube-system)

If you are upgrading from one of the following versions there may be additional steps that you should take.

pre `v1.0.0`

Versions v1.0.0-rc.1 and later use the v1 version of the SecretProviderClass and SecretProviderClassPodStatus CustomResourceDefinitions. secrets-store.csi.x-k8s.io/v1alpha1 versioned CRDs will continue to work, but consider updating your YAMLs to secrets-store.csi.x-k8s.io/v1.

pre `v0.3.0`

The helm chart repository URL has changed to https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts.

Run the following commands to update your Helm chart repositories:

helm repo rm secrets-store-csi-driver
helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm repo update

pre `v0.1.0`

NOTE: CustomResourceDefinitions (CRDs) have been moved from templates to crds directory in the helm charts. To manage the lifecycle of the CRDs during install/upgrade, helm pre-install and pre-upgrade hook has been added. This hook will create a pod that runs only on linux nodes and deploys the CRDs in the Kubernetes cluster.

In case there is an issue with these hooks we recommend backing up your SecretProviderClasses in case of any issues with the hooks:

kubectl get secretproviderclass -A -o yaml > spc-all-backup.yaml

The filtered watch feature is enabled by default in v0.1.0 (see #550). All existing nodePublishSecretRef Kubernetes Secrets used in volume mounts must have the secrets-store.csi.k8s.io/used=true label otherwise secret rotations will fail with failed to get node publish secret errors.

Label these Kubernetes Secrets by running:

kubectl label secret <node publish secret ref name> secrets-store.csi.k8s.io/used=true

pre `v0.0.23`

v0.0.23 sets syncSecret.enabled=false by default. This means the RBAC clusterrole and clusterrolebinding required for sync mounted content as Kubernetes secret will no longer be created by default as part of helm install/upgrade. If you’re using the driver to sync mounted content as Kubernetes secret, you’ll need to set syncSecret.enabled=true as part of helm install/upgrade.

pre `v0.0.20`

v0.0.20 removed support for non-gRPC based providers. Follow your provider documentation to upgrade providers to use gRPC before upgrading the driver to v0.0.20 or greater.

pre `v0.0.18`

v0.0.17 and earlier installed the driver to the default namespace when using the YAML based install. Newer versions of the driver YAML files install the driver to the kube-system namespace. After applying the new YAML files to your cluster run the following to clean up old resources:

kubectl delete daemonset csi-secrets-store --namespace=default
kubectl delete daemonset csi-secrets-store-windows --namespace=default
kubectl delete serviceaccount secrets-store-csi-driver --namespace=default

pre `v0.0.12`

The SecretProviderClass needs to be in the same namespace as the pod referencing it as of v0.0.12.

Defining driver configuration and provider-specific parameters to the CSI driver in pod.Spec[].Volumes has been deprecated in v0.0.12. It is now mandatory to use SecretProviderClass custom resource.

Topics

This section contains information about various features supported by Secrets Store CSI Driver.

Command Reference

The Secrets Store CSI Driver can be provided with the following command line arguments.

The secrets-store container in the DaemonSet can be configured using the following command line arguments:

List of command line options

Parameter	Description	Default
`--endpoint`	CSI endpoint	`unix://tmp/csi.sock`
`--drivername`	Name of the driver	`secrets-store.csi.k8s.io`
`--nodeid`	Node ID
`--log-format-json`	Set log formatter to json	`false`
`--provider-volume`	Volume path for provider	`/etc/kubernetes/secrets-store-csi-providers`
`--additional-provider-volume-paths`	Comma separated list of additional paths to communicate with providers	`/var/run/secrets-store-csi-providers`
`--metrics-addr`	The address the metric endpoint binds to	`:8095`
`--enable-secret-rotation`	Enable secret rotation feature [alpha]	`false`
`--rotation-poll-interval`	Secret rotation poll interval duration	`2m`
`--enable-pprof`	Enable pprof profiling	`false`
`--pprof-port`	Port for pprof profiling	`6065`
`--max-call-recv-msg-size`	Maximum size in bytes of gRPC response from plugins	`4194304`
`--provider-health-check`	Enable health check for configured providers	`false`
`--provider-health-check-interval`	Provider healthcheck interval duration	`2m`

Metrics provided by Secrets Store CSI Driver

The Secrets Store CSI Driver uses opentelemetry for reporting metrics. This project is under active development

Prometheus is the only exporter that’s currently supported with the driver.

List of metrics provided by the driver

Metric	Description	Tags
total_node_publish	Total number of successful volume mount requests	`os_type=<runtime os>`
`provider=<provider name>`
total_node_unpublish	Total number of successful volume unmount requests	`os_type=<runtime os>`
total_node_publish_error	Total number of errors with volume mount requests	`os_type=<runtime os>`
`provider=<provider name>`
`error_type=<error code>`
total_node_unpublish_error	Total number of errors with volume unmount requests	`os_type=<runtime os>`
total_sync_k8s_secret	Total number of k8s secrets synced	`os_type=<runtime os>`
`provider=<provider name>`
sync_k8s_secret_duration_sec	Distribution of how long it took to sync k8s secret	`os_type=<runtime os>`
total_rotation_reconcile	Total number of rotation reconciles	`os_type=<runtime os>`
`rotated=<true or false>`
total_rotation_reconcile_error	Total number of rotation reconciles with error	`os_type=<runtime os>`
`rotated=<true or false>`
`error_type=<error code>`
rotation_reconcile_duration_sec	Distribution of how long it took to rotate secrets-store content for pods	`os_type=<runtime os>`

Metrics are served from port 8095, but this port is not exposed outside the pod by default. Use kubectl port-forward to access the metrics over localhost:

kubectl port-forward ds/csi-secrets-store -n kube-system 8095:8095 &
curl localhost:8095/metrics

Sample Metrics output

# HELP sync_k8s_secret_duration_sec Distribution of how long it took to sync k8s secret
# TYPE sync_k8s_secret_duration_sec histogram
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="0.1"} 0
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="0.2"} 0
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="0.3"} 0
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="0.4"} 1
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="0.5"} 1
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="1"} 1
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="1.5"} 1
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="2"} 1
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="2.5"} 1
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="3"} 1
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="5"} 1
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="10"} 1
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="15"} 1
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="30"} 1
sync_k8s_secret_duration_sec_bucket{os_type="linux",le="+Inf"} 1
sync_k8s_secret_duration_sec_sum{os_type="linux"} 0.3115892
sync_k8s_secret_duration_sec_count{os_type="linux"} 1
# HELP total_node_publish Total number of node publish calls
# TYPE total_node_publish counter
total_node_publish{os_type="linux",provider="azure"} 1
# HELP total_node_publish_error Total number of node publish calls with error
# TYPE total_node_publish_error counter
total_node_publish_error{error_type="ProviderBinaryNotFound",os_type="linux",provider="azure"} 2
total_node_publish_error{error_type="SecretProviderClassNotFound",os_type="linux",provider=""} 4
# HELP total_node_unpublish Total number of node unpublish calls
# TYPE total_node_unpublish counter
total_node_unpublish{os_type="linux"} 1
# HELP total_sync_k8s_secret Total number of k8s secrets synced
# TYPE total_sync_k8s_secret counter
total_sync_k8s_secret{os_type="linux",provider="azure"} 1

Auto rotation of mounted contents and synced Kubernetes Secrets

Design doc: Rotation Design
Feature State: Secrets Store CSI Driver v0.0.15 [alpha]

When the secret/key is updated in external secrets store after the initial pod deployment, the updated secret will be periodically updated in the pod mount and the Kubernetes Secret.

Depending on how the application consumes the secret data:

Mount Kubernetes secret as a volume: Use auto rotation feature + Sync K8s secrets feature in Secrets Store CSI Driver, application will need to watch for changes from the mounted Kubernetes Secret volume. When the Kubernetes Secret is updated by the CSI Driver, the corresponding volume contents are automatically updated.
Application reads the data from container’s filesystem: Use rotation feature in Secrets Store CSI Driver, application will need to watch for the file change from the volume mounted by the CSI driver.
Using Kubernetes secret for environment variable: The pod needs to be restarted to get the latest secret as environment variable.
1. Use something like Reloader to watch for changes on the synced Kubernetes secret and do rolling upgrades on pods

Enable auto rotation

NOTE: This alpha feature is not enabled by default.

To enable auto rotation, enable the --enable-secret-rotation feature gate for the secrets-store container in the Secrets Store CSI Driver pods. The rotation poll interval can be configured using --rotation-poll-interval. The default rotation poll interval is 2m. If using helm to install the driver, set enableSecretRotation: true and configure the rotation poll interval by setting rotationPollInterval. The rotation poll interval can be tuned based on how frequently the mounted contents for all pods and Kubernetes secrets need to be resynced to the latest.

The Secrets Store CSI Driver will update the pod mount and the Kubernetes Secret defined in secretObjects of SecretProviderClass periodically based on the rotation poll interval to the latest value.
If the SecretProviderClass is updated after the pod was initially created
- Adding/deleting objects and updating keys in existing secretObjects - the pod mount and Kubernetes secret will be updated with the new objects added to the SecretProviderClass.
- Adding new secretObject to the existing secretObjects - the Kubernetes secret will be created by the controller.

How to view the current secret versions loaded in pod mount

The Secrets Store CSI Driver creates a custom resource SecretProviderClassPodStatus to track the binding between a pod and SecretProviderClass. This SecretProviderClassPodStatus status also contains the details about the secrets and versions currently loaded in the pod mount.

The SecretProviderClassPodStatus is created in the same namespace as the pod with the name <pod name>-<namespace>-<secretproviderclass name>

➜ kubectl get secretproviderclasspodstatus nginx-secrets-store-inline-crd-default-azure-spc -o yaml
...
status:
  mounted: true
  objects:
  - id: secret/secret1
    version: b82206cb5ac249918008b0b97fd1fd66
  - id: key/key1
    version: 7cc095105411491b84fe1b92ebbcf01a
  podName: nginx-secrets-store-inline-multiple-crd
  secretProviderClassName: azure-spc
  targetPath: /var/lib/kubelet/pods/1b7b0740-62d5-4776-a0df-90d060ef35ba/volumes/kubernetes.io~csi/secrets-store-inline-0/mount

Limitations

The auto rotation feature is only supported with providers that have implemented gRPC server for enabling driver-provider communication.

Sync as Kubernetes Secret

Examples

SecretProviderClass

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-sync
spec:
  provider: azure
  secretObjects:                                 # [OPTIONAL] SecretObject defines the desired state of synced K8s secret objects
  - secretName: foosecret
    type: Opaque
    labels:
      environment: "test"
    data:
    - objectName: secretalias                    # name of the mounted content to sync. this could be the object name or object alias
      key: username
  parameters:
    usePodIdentity: "true"
    keyvaultName: "$KEYVAULT_NAME"               # the name of the KeyVault
    objects: |
      array:
        - |
          objectName: $SECRET_NAME
          objectType: secret                     # object types: secret, key or cert
          objectAlias: secretalias
          objectVersion: $SECRET_VERSION         # [OPTIONAL] object versions, default to latest if empty
        - |
          objectName: $KEY_NAME
          objectType: key
          objectVersion: $KEY_VERSION
    tenantId: "tid"                             # the tenant ID of the KeyVault

Pod yaml

kind: Pod
apiVersion: v1
metadata:
  name: secrets-store-inline
spec:
  containers:
    - name: busybox
      image: registry.k8s.io/e2e-test-images/busybox:1.29
      command:
      - "/bin/sleep"
      - "10000"
      volumeMounts:
      - name: secrets-store01-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
  volumes:
    - name: secrets-store01-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "azure-sync"

In some cases, you may want to create a Kubernetes Secret to mirror the mounted content. Use the optional secretObjects field to define the desired state of the synced Kubernetes secret objects. The volume mount is required for the Sync With Kubernetes Secrets

NOTE: If the provider supports object alias for the mounted file, then make sure the objectName in secretObjects matches the name of the mounted content. This could be the object name or the object alias.

The secrets will only sync once you start a pod mounting the secrets. Solely relying on the syncing with Kubernetes secrets feature thus does not work. When all the pods consuming the secret are deleted, the Kubernetes secret is also deleted.

A SecretProviderClass custom resource should have the following components:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: my-provider
spec:
  provider: vault                             # accepted provider options: azure or vault or gcp
  secretObjects:                              # [OPTIONAL] SecretObject defines the desired state of synced K8s secret objects
  - data:
    - key: username                           # data field to populate
      objectName: foo1                        # name of the mounted content to sync. this could be the object name or the object alias
    secretName: foosecret                     # name of the Kubernetes Secret object
    type: Opaque                              # type of the Kubernetes Secret object e.g. Opaque, kubernetes.io/tls

NOTE: Here is the list of supported Kubernetes Secret types: Opaque, kubernetes.io/basic-auth, bootstrap.kubernetes.io/token, kubernetes.io/dockerconfigjson, kubernetes.io/dockercfg, kubernetes.io/ssh-auth, kubernetes.io/service-account-token, kubernetes.io/tls.

Set as ENV var

Examples

SecretProviderClass

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-sync
spec:
  provider: azure
  secretObjects:                                 # [OPTIONAL] SecretObject defines the desired state of synced K8s secret objects
  - secretName: foosecret
    type: Opaque
    labels:
      environment: "test"
    data:
    - objectName: secretalias                    # name of the mounted content to sync. this could be the object name or object alias
      key: username
  parameters:
    usePodIdentity: "false"
    keyvaultName: "$KEYVAULT_NAME"               # the name of the KeyVault
    objects: |
      array:
        - |
          objectName: $SECRET_NAME
          objectType: secret                     # object types: secret, key or cert
          objectAlias: secretalias
          objectVersion: $SECRET_VERSION         # [OPTIONAL] object versions, default to latest if empty
        - |
          objectName: $KEY_NAME
          objectType: key
          objectVersion: $KEY_VERSION
    tenantId: "tid"                             # the tenant ID of the KeyVault

Pod yaml

kind: Pod
apiVersion: v1
metadata:
  name: secrets-store-inline
spec:
  containers:
  - name: busybox
    image: registry.k8s.io/e2e-test-images/busybox:1.29
    command:
    - "/bin/sleep"
    - "10000"
    volumeMounts:
    - name: secrets-store01-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
    env:
    - name: SECRET_USERNAME
      valueFrom:
        secretKeyRef:
          name: foosecret
          key: username
  volumes:
  - name: secrets-store01-inline
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "azure-sync"

Once the secret is created, you may wish to set an ENV VAR in your deployment to reference the new Kubernetes secret.

spec:
  containers:
  - image: registry.k8s.io/e2e-test-images/busybox:1.29
    name: busybox
    command:
    - "/bin/sleep"
    - "10000"
    env:
    - name: SECRET_USERNAME
      valueFrom:
        secretKeyRef:
          name: foosecret
          key: username

Best Practices

Deploy the driver and providers into the kube-system or a separate dedicated namespace.

The driver is installed as a DaemonSet with the ability to mount kubelet hostPath volumes and view pod service account tokens. It should be treated as privileged and regular cluster users should not have permissions to deploy or modify the driver.
Do not grant regular cluster users permissions to modify SecretProviderClassPodStatus resources.

The SecretProviderClassPodStatus CRD is used by the driver to keep track of mounted resources. Manually editing this resource could have unexpected consequences to the system health and in particular modifying SecretProviderClassPodStatus/status may have security implications.
Disable Secret sync if not needed.

If you do not intend to use the Secret syncing feature, do not install the RBAC permissions that allow the driver to access cluster Secret objects.

This can be done by setting syncSecret.enabled = false when installing with helm.
Enable KMS application wrapping if using Secret sync.

If you need to synchronise your external secrets to Kubernetes Secrets consider configuring encryption of data at rest

This will ensure that data is encrypted before it is stored in etcd.
Keep the driver up to date.

Subscribe to the kubernetes-secrets-store-csi-driver mailing list to be notified of new releases and security announcements.

Consider using the Github Watch feature to subscribe to releases as well.

Always be sure to review the release notes before upgrading.
When evaluating this driver consider the following threats:
- When a secret is accessible on the filesystem, application vulnerabilities like directory traversal attacks can become higher severity as the attacker may gain the ability read the secret material.
- When a secret is consumed through environment variables, misconfigurations such as enabling a debug endpoints or including dependencies that log process environment details may leak secrets.
- When syncing secret material to Kubernetes Secrets, consider whether the access controls on that data store are sufficiently narrow in scope. If possible, directly integrating with a purpose built secrets API may offer the best security tradeoffs.

Providers

This project features a pluggable provider interface developers can implement that defines the actions of the Secrets Store CSI driver. This enables retrieval of sensitive objects stored in an enterprise-grade external secrets store into Kubernetes while continue to manage these objects outside of Kubernetes.

Criteria for Supported Providers

Here is a list of criteria for supported provider:

Code audit of the provider implementation to ensure it adheres to the required provider-driver interface - Implementing a Provider for Secrets Store CSI Driver
Add provider to the e2e test suite to demonstrate it functions as expected. Please use existing providers e2e tests as a reference.
If any update is made by a provider (not limited to security updates), the provider is expected to update the provider’s e2e test in this repo.

Removal from Supported Providers

Failure to adhere to the Criteria for Supported Providers will result in the removal of the provider from the supported list and subject to another review before it can be added back to the list of supported providers.

When a provider’s e2e tests are consistently failing with the latest version of the driver, the driver maintainers will coordinate with the provider maintainers to provide a fix. If the test failures are not resolved within 4 weeks, then the provider will be removed from the list of supported providers.

Implementing a Provider for Secrets Store CSI Driver

This document highlights the implementation steps for adding a secrets-store-csi-driver provider.

Implementation details

The driver uses gRPC to communicate with the provider. To implement a secrets-store-csi-driver provider, you can develop a new provider gRPC server using the stub file available for Go.

Use the functions and data structures in the stub file: service.pb.go to develop the server code
- The stub file and proto file are shared and hosted in the driver. Vendor-in the stub file and proto file in the provider
- fake server example
Provider runs as a daemonset and is deployed on the same host(s) as the secrets-store-csi-driver pods
Provider Unix Domain Socket volume path. The default volume path for providers is /etc/kubernetes/secrets-store-csi-providers. Add the Unix Domain Socket to the dir in the format /etc/kubernetes/secrets-store-csi-providers/<provider name>.sock
The <provider name> in <provider name>.sock must match the regular expression ^[a-zA-Z0-9_-]{0,30}$

See design doc for more details.

Features supported by current providers

Features \ Providers	Azure	GCP	AWS	Vault	Akeyless	Conjur
Sync as Kubernetes secret	Yes	Yes	Yes	Yes	Yes	Yes
Rotation	Yes	Yes	Yes	Yes	Yes	Yes
Windows	Yes	No	No	No	No	No
Helm Chart	Yes	No	Yes	Yes	Yes	Yes

Troubleshooting

An overview of a list of components to assist in troubleshooting.

Logging

To troubleshoot issues with the csi driver, you can look at logs from the secrets-store container of the csi driver pod running on the same node as your application pod:

kubectl get pod -o wide
# find the secrets store csi driver pod running on the same node as your application pod

kubectl logs csi-secrets-store-secrets-store-csi-driver-7x44t secrets-store

If the pod fails to start because of the inline volume mount, you can describe the pod to view mount failure errors and events:

kubectl describe pod <application pod>

It is always a good idea to include relevant logs from csi driver pod when opening a new issue.

pprof

Starting the secrets-store container in driver with --enable-pprof=true will enable a debug http endpoint at --pprof-port (default: 6065). Accessing this will also require port-forward:

kubectl port-forward csi-secrets-store-secrets-store-csi-driver-7x44t secrets-store 6065:6065 &
curl localhost:6065/debug/pprof

Common Errors

`SecretProviderClass` not found

kubectl describe pod <application pod> shows:

  Warning  FailedMount  3s (x4 over 6s)  kubelet, kind-control-plane  MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to get secretproviderclass default/azure, error: secretproviderclasses.secrets-store.csi.x-k8s.io "azure" not found

The SecretProviderClass being referenced in the volumeMount needs to exist in the same namespace as the application pod.

Volume mount fails with `secrets-store.csi.k8s.io` not found in the list of registered CSI drivers

kubectl describe pod <application pod> shows:

  Warning  FailedMount  1s (x4 over 4s)  kubelet, kind-control-plane  MountVolume.SetUp failed for volume "secrets-store-inline" : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name secrets-store.csi.k8s.io not found in the list of registered CSI drivers

Secrets Store CSI Driver is deployed as a DaemonSet. The above error indicates the CSI driver pods aren’t running on the node.

If the node is tainted, then add a toleration for the taint in the Secrets Store CSI Driver DaemonSet.
Check to see if there are any node selectors preventing the Secrets Store CSI Driver pods from running on the node.

Check to see if the CSIDriver object has been deployed to the cluster:

# This is the desired output. If the secrets-store.csi.k8s.io isn't found, then reinstall the driver.
kubectl get csidriver
NAME                       ATTACHREQUIRED   PODINFOONMOUNT   MODES       AGE
secrets-store.csi.k8s.io   false            true             Ephemeral   110m

Mount fails with `grpc: received message larger than max`

If the files pulled in by the SecretProviderClass are larger than 4MiB you may observe FailedMount warnings with a message that includes grpc: received message larger than max. You can configure the driver to accept responses larger than 4MiB by specifying the --max-call-recv-msg-size=<size in bytes> argument to the secrets-store container in the csi-secrets-store DaemonSet.

Note that this may also increase memory resource consumption of the secrets-store container, so you should also consider increasing the memory limit as well.

failed to get CSI client: `driver name secrets-store.csi.k8s.io not found in the list of registered CSI drivers`

Volume mount fails with `"GRPC error" err="failed to mount objects, error: failed to write file: no such file or directory`

Some Kubernetes distros (such as Rancher and Microk8s) use a custom kubeletRootDir path. This may cause errors such as volume mount failures or failures to register CSI drivers. If the default kubelet directory path of the distro you are using is not /var/lib/kubelet, it can be configured during installation via Helm chart using --set linux.kubeletRootDir=<desired/kubelet/dir/path>. For Rancher the kubelet directory path is /opt/rke/var/lib/kubelet and for Microk8s it is /var/snap/microk8s/common/var/lib/kubelet.

Load tests

This document highlights the results from load tests using secrets-store-csi-driver v0.0.21.

Note: Refer to doc for more details on the optimization done as part of v0.0.21 release.

The results posted here can be used as a guide for configuring resource and memory limits for the CSI driver daemonset pods.

Testing Environment

250 nodes Azure Kubernetes Service cluster
- VM Size: Standard_DS2_v2 (2 vCPU, 7GiB)
3500 Kubernetes secrets in the cluster
- These secrets were pre-configured to ensure existing Kubernetes secrets doesn’t impact the memory for the CSI driver.
7250 pods running in the cluster
- These pods were pre-configured to ensure existing Kubernetes pods doesn’t impact the memory for the CSI driver.

The Secrets Store CSI Driver and Azure Keyvault Provider were deployed to the cluster.

Secrets Store CSI Driver features enabled:

Sync as Kubernetes secret
Secret Auto rotation
- Rotation Poll Interval: 2m

Testing scenarios

10000 pods with CSI volume

10000 pods with CSI volume.
- Total number of pods in the cluster = 7250 + 10000 = 17250 pods.
SecretProviderClass with syncing 2 Kubernetes secrets.

➜ kubectl top pods -l app=csi-secrets-store -n kube-system --sort-by=memory
NAME                      CPU(cores)   MEMORY(bytes)
csi-secrets-store-kd2bc   3m           54Mi
csi-secrets-store-wx6z9   3m           52Mi
csi-secrets-store-6gjqq   3m           52Mi
csi-secrets-store-knl5g   4m           52Mi
csi-secrets-store-9lzzn   4m           51Mi

The current default memory and resource limits have been configured based on the above tests.

Understanding Secrets Store CSI Driver memory consumption

As of Secrets Store CSI Driver v0.0.21, the memory consumption for the driver is dependent on:

Number of pods on the same node as the driver pod.
Number of secrets with a. secrets-store.csi.k8s.io/managed=true label. This label is set for all the secrets created by the Secrets Store CSI Driver. b. secrets-store.csi.k8s.io/used=true label. This label needs to be set for all nodePublishSecretRef.
Number of SecretProviderClass across all namespaces.
Number of SecretProviderClassPodStatus created by Secrets Store CSI Driver for the pod on the same node as the application pod.
1. Secrets Store CSI Driver creates a SecretProviderClassPodStatus to map pod to SecretProviderClass. See doc for more details.

Testing

Unit Tests

Run unit tests locally with make test.

End-to-end Tests

End-to-end tests automatically runs on Prow when a PR is submitted. If you want to run using a local or remote Kubernetes cluster, make sure to have kubectl, helm and bats set up in your local environment and then run make e2e-azure, make e2e-vault or make e2e-gcp with custom images.

Job config for test jobs run for each PR in prow can be found here

Known Limitations

This document highlights the current limitations when using secrets-store-csi-driver.

Mounted content and Kubernetes Secret not updated

When the secret/key is updated in external secrets store after the initial pod deployment, the updated secret is not automatically reflected in the pod mount or the Kubernetes secret.
When the SecretProviderClass is updated after the pod was initially created.
Adding/deleting objects and updating keys in existing secretObjects doesn’t result in update of Kubernetes secrets.

The CSI driver is invoked by kubelet only during the pod volume mount. So subsequent changes in the SecretProviderClass after the pod has started doesn’t trigger an update to the content in volume mount or Kubernetes secret.

Enable Secret autorotation feature has been released in v0.0.15+. Refer to doc and design doc for more details.

How to fetch the latest content with release `v0.0.14` and earlier or without `Auto rotation` feature enabled?

If the SecretProviderClass has secretObjects defined, then delete the Kubernetes secret.
Restart the application pod.

When the pod is recreated, kubelet invokes the CSI driver for mounting the volume. As part of this mount request, the latest content will be fetched from external secrets store and populated in the pod. The same content is then mirrored in the Kubernetes secret data.

Secrets not rotated when using subPath volume mount

A container using subPath volume mount will not receive secret updates when it is rotated.

    volumeMounts:
    - mountPath: /app/spapi/settings.ini
      name: app-config
      subPath: settings.ini
...
  volumes:
  - csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: app-config
    name: app-config

Secrets Store CSI Driver uses atomic writer to write the secret files. This is the same writer used by Kubernetes to write secret, configmap and downward API volumes. Atomic writer relies on symlinks to update the content of the file. The secret file is bind mounted into the container and is a symlink to the actual secret file in a timestamped directory. When the secret gets updated, the symlink is updated but the actual secret file bind mounted into the container remains unchanged. Refer to kubernetes/kubernetes#50345 for more details.

The only way to get the latest content is to restart the pod or not use subPath volume mount.

Release Management

Overview

This document describes Kubernetes Secrets Store CSI Driver project release management, which talks about versioning, branching and cadence.

Legend

X.Y.Z refers to the version (git tag) of Secrets Store CSI Driver that is released. This is the version of the Secrets Store CSI Driver image.
Milestone should be designed to include feature sets to accommodate monthly release cycles including test gates. GitHub milestones are used by maintainers to manage each release. PRs and Issues for each release should be created as part of a corresponding milestone.
Test gates should include soak tests and upgrade tests from the last minor version.

Versioning

This project strictly follows semantic versioning. All releases will be of the form vX.Y.Z where X is the major version, Y is the minor version and Z is the patch version.

Patch releases

Patch releases provide users with bug fixes and security fixes. They do not contain new features.

Minor releases

Minor releases contain security and bug fixes as well as new features.
They are backwards compatible.

Major releases

Major releases contain breaking changes. Breaking changes refer to schema changes and behavior changes of Secrets Store CSI Driver that may require a clean install during upgrade and it may introduce changes that could break backward compatibility.
Ideally we will avoid making multiple major releases to be always backward compatible, unless project evolves in important new directions and such release is necessary.

Release Cadence and Branching

Secrets Store CSI Driver follows monthly release schedule.
A new release should be created in second week of each month. This schedule not only allows us to do bug fixes, but also provides an opportunity to address underline image vulnerabilities etc. if any.
The new version is decided as per above guideline and release branch should be created from main with name release-<version>. For eg. release-0.1. Then build the image from release branch.
Any fixes or patches should be merged to main and then cherry pick to the release branch.

Security Vulnerabilities

We use trivy to scan the base image for known vulnerabilities. When a vulnerability is detected and has a fixed version, we will update the image to include the fix. For vulnerabilities that are not in a fixed version, there is nothing that can be done immediately. Fixable CVE patches will be part of the patch releases published second week of every month.

Supported Releases

Applicable fixes, including security fixes, may be cherry-picked into the release branch, depending on severity and feasibility. Patch releases are cut from that branch as needed.

We expect users to stay reasonably up-to-date with the versions of Secrets Store CSI Driver they use in production, but understand that it may take time to upgrade. We expect users to be running approximately the latest patch release of a given minor release and encourage users to upgrade as soon as possible.

We expect to “support” n (current) and n-1 major.minor releases. “Support” means we expect users to be running that version in production. For example, when v1.3.0 comes out, v1.1.x will no longer be supported for patches and we encourage users to upgrade to a supported version as soon as possible.

Supported Kubernetes Versions

Secrets Store CSI Driver will maintain support for all actively supported Kubernetes minor releases per Kubernetes Supported Versions policy. If you choose to use Secrets Store CSI Driver with a version of Kubernetes that it does not support, you are using it at your own risk.

Acknowledgement

This document builds on the ideas and implementations of release processes from projects like Gatekeeper, Helm and Kubernetes.

Design Docs

The Secrets Store CSI Driver uses Google Docs for design documents and proposals. This doc is a constant work in progress, subject to frequent revision. Features are listed in no particular order.