Certificate Management

This article assumes you have already using the Public operator or Helm. However, it’s possible to configure these settings before starting CockroachDB on Kubernetes.

By default, self-signed certificates are used when using the Public operator or Helm to securely . However, the recommended approach is to use cert-manager for certificate management. For details, refer to Deploy cert-manager for mTLS.

This page is for Kubernetes deployments that are not using the CockroachDB operator. For guidance specific to the CockroachDB operator, read .

This page describes a Kubernetes deployment method that is considered “legacy” and will not be supported in a future release of CockroachDB. New CockroachDB deployments on Kubernetes are recommended to use the newer, fully-featured that’s easier to deploy and supports scaling of multi-region clusters. To migrate an existing deployment to use the CockroachDB operator, read the and migration guides.

For more information on legacy Kubernetes deployments and support timelines, read the .

This page explains how to:

• Authenticate a Public operator or Helm deployment using a custom CA
• Rotate security certificates
• Secure the webhooks (public perator)

If you are running a secure Helm deployment on Kubernetes 1.22 and later, you must migrate away from using the Kubernetes CA for cluster authentication. The recommended approach is to use cert-manager for certificate management. For details, refer to Deploy cert-manager for mTLS.

All kubectl steps should be performed in the . By default, this is cockroach-operator-system.

Use a custom CA

Public operator

By default, the Public operator will generate and sign 1 client and 1 node certificate to secure the cluster.

To use your own certificate authority instead, add the following to the Public operator’s custom resource before :

spec:
  nodeTLSSecret: {node_secret_name}
  clientTLSSecret: {client_secret_name}

Replace:

• : The name of the Kubernetes secret that contains the generated node certificate and key.
• : The name of the Kubernetes secret that contains the generated client certificate and key.

Currently, the Public operator requires that the client and node secrets each contain the filenames tls.crt and tls.key.

Apply the new settings to the cluster:

$ kubectl apply -f example.yaml

Helm

By default on secure deployments, the Helm chart will generate and sign one client certificate and one node certificate to secure the cluster.

To use your own certificate authority instead, specify the following in the custom values file you created when :

tls:
  enabled: true
  certs:
  provided: true
  nodeSecret: {node_secret_name}
  clientRootSecret: {client_secret_name}

Replace:

• : The name of the Kubernetes secret that contains the generated node certificate and key.
• : The name of the Kubernetes secret that contains the generated client certificate and key.

Apply the custom values to override the default Helm chart values:

$ helm upgrade {release-name} --values {custom-values}.yaml cockroachdb/cockroachdb

Example: Authenticate with cockroach cert

This example uses to generate and sign the CockroachDB node and client certificates. To learn more about the supported methods of signing certificates, refer to .

Public operator

Complete the following steps before .

1. Create two directories:

   mkdir certs my-safe-directory
   

   Directory	Description
   certs	You’ll generate your CA certificate and all node and client certificates and keys in this directory.
   my-safe-directory	You’ll generate your CA key in this directory and then reference the key when generating node and client certificates.

2. Create the CA certificate and key pair:

   cockroach cert create-ca \
     --certs-dir=certs \
     --ca-key=my-safe-directory/ca.key
   

3. Create a client certificate and key pair for the root user:

   cockroach cert create-client root \
     --certs-dir=certs \
     --ca-key=my-safe-directory/ca.key
   

4. Upload the client certificate and key to the Kubernetes cluster as a secret, renaming them to the filenames required by the Public operator:

   kubectl create secret generic cockroachdb.client.root \
     --from-file=tls.key=certs/client.root.key \
     --from-file=tls.crt=certs/client.root.crt \
     --from-file=ca.crt=certs/ca.crt
   
   secret/cockroachdb.client.root created
   

5. Create the certificate and key pair for your CockroachDB nodes, specifying the namespace you used when . This example uses the Public operator’s default namespace (cockroach-operator-system):

   cockroach cert create-node localhost \
     127.0.0.1 \
     cockroachdb-public \
     cockroachdb-public.cockroach-operator-system \
     cockroachdb-public.cockroach-operator-system.svc.cluster.local \
     *.cockroachdb \
     *.cockroachdb.cockroach-operator-system \
     *.cockroachdb.cockroach-operator-system.svc.cluster.local \
     --certs-dir=certs \
     --ca-key=my-safe-directory/ca.key
   

6. Upload the node certificate and key to the Kubernetes cluster as a secret, renaming them to the filenames required by the Public operator:

   kubectl create secret generic cockroachdb.node \
     --from-file=tls.key=certs/node.key \
     --from-file=tls.crt=certs/node.crt \
     --from-file=ca.crt=certs/ca.crt
   
   secret/cockroachdb.node created
   

7. Check that the secrets were created on the cluster:

   kubectl get secrets
   
   NAME                      TYPE                                   DATA   AGE
   cockroachdb.client.root   Opaque                                   3    13s
   cockroachdb.node          Opaque                                   3     3s
   default-token-6js7b       kubernetes.io/service-account-token      3     9h
   

8. Add nodeTLSSecret and clientTLSSecret to the Public operator’s , specifying the generated secret names:

   spec:
     clientTLSSecret: cockroachdb.client.root
     nodeTLSSecret: cockroachdb.node
   

Helm

1. Create two directories:

   mkdir certs my-safe-directory
   

   Directory	Description
   certs	You’ll generate your CA certificate and all node and client certificates and keys in this directory.
   my-safe-directory	You’ll generate your CA key in this directory and then reference the key when generating node and client certificates.

2. Create the CA certificate and key pair:

   cockroach cert create-ca \
     --certs-dir=certs \
     --ca-key=my-safe-directory/ca.key
   

3. Create a client certificate and key pair for the root user:

   cockroach cert create-client root \
     --certs-dir=certs \
     --ca-key=my-safe-directory/ca.key
   

4. Upload the client certificate and key to the Kubernetes cluster as a secret:

   kubectl create secret generic cockroachdb.client.root --from-file=certs
   
   secret/cockroachdb.client.root created
   

5. Create the certificate and key pair for your CockroachDB nodes:

   cockroach cert create-node localhost \
     127.0.0.1 \
     my-release-cockroachdb-public \
     my-release-cockroachdb-public.default \
     my-release-cockroachdb-public.default.svc.cluster.local \
     *.my-release-cockroachdb \
     *.my-release-cockroachdb.default \
     *.my-release-cockroachdb.default.svc.cluster.local \
     --certs-dir=certs \
     --ca-key=my-safe-directory/ca.key
   

This example assumes that you followed our , which uses my-release as the release name. If you used a different value, be sure to adjust the release name in this command.

1. Upload the node certificate and key to the Kubernetes cluster as a secret:

   kubectl create secret generic cockroachdb.node --from-file=certs
   
   secret/cockroachdb.node created
   

2. Check that the secrets were created on the cluster:

   kubectl get secrets
   
   NAME                      TYPE                               DATA   AGE
   cockroachdb.client.root   Opaque                                3   41m
   cockroachdb.node          Opaque                                5   14s
   default-token-6qjdb       kubernetes.io/service-account-token   3    4m
   

3. Specify the following in the custom values file you created when , using the generated secret names:

   tls:
     enabled: true
     certs:
     provided: true
     clientRootSecret: cockroachdb.client.root
     nodeSecret: cockroachdb.node
   

4. Apply the custom values to override the default Helm chart values:

   helm upgrade {release-name} --values {custom-values}.yaml cockroachdb/cockroachdb
   

Rotate security certificates

You may need to rotate the node, client, or CA certificates in the following scenarios:

• The node, client, or CA certificates are expiring soon.
• Your organization’s compliance policy requires periodic certificate rotation.
• The key (for a node, client, or CA) is compromised.
• You need to modify the contents of a certificate, for example, to add another DNS name or the IP address of a load balancer through which a node can be reached. In this case, you would need to rotate only the node certificates.

Example: Rotate certificates signed with cockroach cert

Public operator

If you previously authenticated with cockroach cert, follow these steps to rotate the certificates using the same CA:

1. Create a new client certificate and key pair for the root user, overwriting the previous certificate and key:

   cockroach cert create-client root \
       --certs-dir=certs \
       --ca-key=my-safe-directory/ca.key \
       --overwrite
   

2. Upload the new client certificate and key to the Kubernetes cluster as a new secret, renaming them to the filenames required by the Public operator:

   kubectl create secret generic cockroachdb.client.root.2 \
     --from-file=tls.key=certs/client.root.key \
     --from-file=tls.crt=certs/client.root.crt \
     --from-file=ca.crt=certs/ca.crt
   
   secret/cockroachdb.client.root.2 created
   

3. Create a new certificate and key pair for your CockroachDB nodes, overwriting the previous certificate and key. Specify the namespace you used when . This example uses the Public operator’s default namespace (cockroach-operator-system):

   cockroach cert create-node localhost \
     127.0.0.1 \
     cockroachdb-public \
     cockroachdb-public.cockroach-operator-system \
     cockroachdb-public.cockroach-operator-system.svc.cluster.local \
     *.cockroachdb \
     *.cockroachdb.cockroach-operator-system \
     *.cockroachdb.cockroach-operator-system.svc.cluster.local \
     --certs-dir=certs \
     --ca-key=my-safe-directory/ca.key \
     --overwrite
   

4. Upload the new node certificate and key to the Kubernetes cluster as a new secret, renaming them to the filenames required by the Public operator:

   kubectl create secret generic cockroachdb.node.2 \
     --from-file=tls.key=certs/node.key \
     --from-file=tls.crt=certs/node.crt \
     --from-file=ca.crt=certs/ca.crt
   
   secret/cockroachdb.node.2 created
   

5. Add nodeTLSSecret and clientTLSSecret to the Public operator’s , specifying the new secret names:

   spec:
     clientTLSSecret: cockroachdb.client.root.2
     nodeTLSSecret: cockroachdb.node.2
   

6. Check that the secrets were created on the cluster:

   kubectl get secrets
   
   NAME                        TYPE                              DATA   AGE
   cockroachdb.client.root.2   Opaque                               3    4s
   cockroachdb.node.2          Opaque                               3    1s
   default-token-6js7b         kubernetes.io/service-account-token  3    9h
   

   Remember that nodeTLSSecret and clientTLSSecret in the Public operator’s must specify these secret names. For details, see Use a custom CA.

7. Apply the new settings to the cluster:

   kubectl apply -f example.yaml
   

   The pods will terminate and restart one at a time, using the new certificates.

8. You can observe this process:

   kubectl get pods
   
   NAME                                  READY   STATUS        RESTARTS   AGE
   cockroach-operator-655fbf7847-lvz6x   1/1     Running         0      4h29m
   cockroachdb-0                         1/1     Running         0      4h16m
   cockroachdb-1                         1/1     Terminating     0      4h16m
   cockroachdb-2                         1/1     Running         0        43s
   

9. Delete the existing client secret that is no longer in use:

   kubectl delete secret cockroachdb.client.root
   
   secret "cockroachdb.client.root" deleted
   

10. Delete the existing node secret that is no longer in use:

    kubectl delete secret cockroachdb.node
    
    secret "cockroachdb.node" deleted
    

Helm

The Helm chart includes values to configure a Kubernetes cron job that regularly rotates certificates before they expire.

If you previously authenticated with cockroach cert, follow these steps to ensure the certificates are rotated:

1. Upload the CA certificate that you previously created to the Kubernetes cluster as a secret:

   kubectl create secret generic cockroachdb.ca \
     --from-file=certs/ca.crt
   
   secret/cockroachdb.ca created
   

2. Specify the following in the custom values file you created when , using the generated secret name:

   selfSigner:
     enabled: true
     caProvided: true
     caSecret: cockroachdb.ca
     rotateCerts: true
   

selfSigner.enabled and selfSigner.rotateCerts are true by default in the Helm chart values.

1. Customize the following selfSigner fields to set the frequency of certificate rotation. These should correspond to the durations of the CA, client, and node certificates.

   selfSigner:
   minimumCertDuration: 624h
   caCertDuration: 43800h
   caCertExpiryWindow: 648h
   clientCertDuration: 672h
   clientCertExpiryWindow: 48h
   nodeCertDuration: 8760h
   nodeCertExpiryWindow: 168h
   

• caCertDuration, clientCertDuration, and nodeCertDuration specify the duration in hours of the CA, client, and node certificates, respectively.
• caCertExpiryWindow, clientCertExpiryWindow, and nodeCertExpiryWindow specify the timeframe in hours during which the CA, client, and node certificates, respectively, should be rotated before they expire.
• minimumCertDuration specifies the minimum duration in hours for all certificates. This is to ensure that the client and node certificates are rotated within the duration of the CA certificate. This value must be less than:

  • cacertExpiryWindow

  • The difference of clientCertDuration and clientExpiryWindow

  • The difference of nodeCertDuration and nodeCertExpiryWindow

    Certificate duration is configured when running . You can check the expiration dates of the cockroach cert certificates by running:

    cockroach cert list --certs-dir=certs
    

    For each certificate, the output includes its certificate file and expiration, the key file for node and client certificates, a Notes column with additional details, and an Errors column that is empty unless there is an error.

1. Apply the custom values to override the default Helm chart values:

   helm upgrade {release-name} --values {custom-values}.yaml cockroachdb/cockroachdb
   

   The certificates will be rotated during the specified expiry windows.

Deploy cert-manager for mTLS

Cockroach Labs recommends using cert-manager to sign certificates for cluster authentication. cert-manager manages certificates and certificate issuers as resource types in Kubernetes clusters, to simplify the process of obtaining, renewing and using those certificates.

Previously, the Helm chart used a self-signer for cluster authentication. This approach is no longer recommended.

1. Install a supported version of cert-manger. For a new cluster, Cockroach Labs recommends using the latest supported version. Refer to installed you will find it cert-manager Installation in the cert-manager project’s documentation.

2. Create a file named issuer.yaml that configures an Issuer, which represents a certificate authority that can sign certificates. This example creates an issuer that can sign self-signed CA certificates. To customize your issuer, refer to Issuer Configuration in the cert-manager project’s documentation.

   apiVersion: cert-manager.io/v1
   kind: Issuer
   metadata:
   name: cockroachdb
   spec:
   selfSigned: {}
   

3. Use kubectl apply to create the issuer from the YAML file:

   kubectl apply -f issuer.yaml
   

4. Enable and configure cert-manager in the Helm chart’s values.yaml file. The following options are required. For more options, refer to cert-manager in the CockroachDB Helm chart documentation.

   tls.certs.selfSigner.enabled: false
   tls.certs.certManager: true
   tls.certs.certManagerIssuer.kind: Issuer
   tls.certs.certManagerIssuer.name: cockroachdb
   

• To disable signing self-signed certificates, set tls.certs.selfSigner.enabled to false.
• Set tls.certs.certManagerIssuer.kind to either Issuer or ClusterIssuer. To get started, Issuer is recommended. ClusterIssuer is cluster-scoped; when referencing a secret via the secretName field, only secrets in the cluster-resource namespace (cert-manager by default) are searched. To learn more, refer to Cluster Resource Namespace in the cert-manager project’s documentation.
• Set certManagerIssuer.name to the name of the issuer you created in the previous step.

1. Apply the updated Helm chart:

   helm install my-release --values values.yaml cockroachdb/cockroachdb
   

   Replace values.yaml with the name of your Helm chart’s values file.

Public operator

Secure the webhooks

The Public operator ships with both mutating and validating webhooks. Communication between the Kubernetes API server and the webhook service must be secured with TLS.

By default, the Public operator searches for the TLS secret cockroach-operator-webhook-ca, which contains a CA certificate. If the secret is not found, the Public operator auto-generates cockroach-operator-webhook-ca with a CA certificate for future runs.

The Public operator then generates a one-time server certificate for the webhook server that is signed with cockroach-operator-webhook-ca. Finally, the CA bundle for both mutating and validating webhook configurations is patched with the CA certificate.

You can also use your own certificate authority rather than cockroach-operator-webhook-ca. Both the certificate and key files you generate must be PEM-encoded. See the following example.

Example: Using OpenSSL to secure the webhooks

These steps demonstrate how to use the openssl genrsa and openssl req subcommands to secure the webhooks on a running Kubernetes cluster:

1. Generate a 4096-bit RSA private key:

   openssl genrsa -out tls.key 4096
   

2. Generate an X.509 certificate, valid for 10 years. You will be prompted for the certificate field values.

   openssl req -x509 -new -nodes -key tls.key -sha256 -days 3650 -out tls.crt
   

3. Create the secret, making sure that :

   kubectl create secret tls cockroach-operator-webhook-ca --cert=tls.crt --key=tls.key
   
   secret/cockroach-operator-webhook-ca created
   

4. Remove the certificate and key from your local environment:

   rm tls.crt tls.key
   

5. Roll the Public operator deployment to ensure a new server certificate is generated:

   kubectl rollout restart deploy/cockroach-operator-manager
   
   deployment.apps/cockroach-operator-manager restarted