Orchestrate CockroachDB Across Multiple Kubernetes Clusters

This page shows you how to orchestrate a secure CockroachDB deployment across three Kubernetes clusters, each in a different geographic region, using StatefulSets to manage the containers within each cluster and linking them together via DNS. This will result in a single, multi-region CockroachDB cluster running on Kubernetes.

This page describes a Kubernetes deployment method that is considered “legacy” and will not be supported in a future release of CockroachDB. New CockroachDB deployments on Kubernetes are recommended to use the newer, fully-featured that’s easier to deploy and supports scaling of multi-region clusters. To migrate an existing deployment to use the CockroachDB operator, read the and migration guides.

For more information on legacy Kubernetes deployments and support timelines, read the .

To deploy CockroachDB in a single Kubernetes cluster instead, see . Also, for details about potential performance bottlenecks to be aware of when running CockroachDB in Kubernetes and guidance on how to optimize your deployment for better performance, see .

Before you begin

Before getting started, it’s helpful to review some Kubernetes-specific terminology and current limitations.

• Kubernetes terminology
• UX differences from running in a single cluster
• Best practices

Kubernetes terminology

Feature	Description
node	A physical or virtual machine. In this tutorial, you’ll run GKE or EKS instances and join them as worker nodes in three independent Kubernetes clusters, each in a different region.
pod	A pod is a group of one or more Docker containers. In this tutorial, each pod will run on a separate GKE or EKS instance and include one Docker container running a single CockroachDB node. You’ll start with 3 pods in each region and grow to 4.
StatefulSet	A group of pods treated as stateful units, where each pod has distinguishable network identity and always binds back to the same persistent storage on restart. StatefulSets are considered stable as of Kubernetes version 1.9 after reaching beta in version 1.5.
persistent volume	A piece of networked storage (Persistent Disk on GCE, Elastic Block Store on AWS) mounted into a pod. The lifetime of a persistent volume is decoupled from the lifetime of the pod that’s using it, ensuring that each CockroachDB node binds back to the same storage on restart. This tutorial assumes that dynamic volume provisioning is available. When that is not the case, persistent volume claims need to be created manually.
CSR	A CSR, or certificate signing request, is a request to have a TLS certificate verified by a certificate authority (CA). A CSR is issued for the CockroachDB node running in each pod, as well as each client as it connects to the Kubernetes cluster.
RBAC	RBAC, or role-based access control, is the system Kubernetes uses to manage permissions within the cluster. In order to take an action (e.g., get or create) on an API resource (e.g., a pod), the client must have a Role that allows it to do so.
namespace	A namespace provides a scope for resources and names within a Kubernetes cluster. Names of resources must be unique within a namespace, but not across namespaces. Most Kubernetes client commands will use the default namespace by default, but can operate on resources in other namespaces as well. In this tutorial, CockroachDB pods will be deployed in their own namespace in each Kubernetes cluster.
kubectl	kubectl is the command-line interface for running commands against Kubernetes clusters.
kubectl context	When multiple Kubernetes clusters are deployed on your account, kubectl “context” specifies a cluster to connect to.

UX differences from running in a single cluster

These instructions create 3 StatefulSets that each run 3 CockroachDB pods in a separate Kubernetes cluster deployed in its own region. If you haven’t often worked with multiple Kubernetes clusters, remember that kubectl commands are run against a cluster in a specific context. Either run kubectl use-context <cluster-context> frequently to switch contexts between commands, or append --context <cluster-context> to the commands you run to ensure they are run against the correct cluster.

Each Kubernetes cluster’s DNS server is pointed at the other clusters’ DNS servers so that DNS lookups for certain zone-scoped suffixes (e.g., *.us-west1-a.svc.cluster.local) can be deferred to the appropriate cluster’s DNS server. To make this work, we create the StatefulSets in namespaces named after the region in which each Kubernetes cluster is deployed. To run a command against one of the pods, append --namespace <cluster-namespace> to your commands. Alternatively, run kubectl config set-context <context-name> --namespace <namespace-name> to set the default namespace for a context.

Because the CockroachDB pods run in a non-default namespace, client applications wanting to talk to CockroachDB from the default namespace would need to use a zone-scoped service name (e.g., cockroachdb-public.us-west1-a) rather than cockroachdb-public, as in a single-cluster setting. However, the setup script used by these instructions sets up an additional ExternalName service in the default namespace such that the clients in the default namespace can simply talk to the cockroachdb-public address.

Best practices

Kubernetes version

To deploy CockroachDB v26.3, Kubernetes 1.18 or higher is required. Cockroach Labs strongly recommends that you use a Kubernetes version that is eligible for patch support by the Kubernetes project.

Public operator

• The Public operator deploys clusters in a single region. For multi-region deployments using manual configs, Cockroach Labs recommends using the which is designed to support multi-region deployments. For guidance on how to force multi-region support with the Public operator, see .

• Using the Public operator, you can give a new cluster an arbitrary number of labels. However, a cluster’s labels cannot be modified after it is deployed. To track the status of this limitation, refer to #993 in the Public operator project’s issue tracker.

Helm version

The CockroachDB Helm chart requires Helm 3.0 or higher. If you attempt to use an incompatible Helm version, an error like the following occurs:

Error: UPGRADE FAILED: template: cockroachdb/templates/tests/client.yaml:6:14: executing "cockroachdb/templates/tests/client.yaml" at <.Values.networkPolicy.enabled>: nil pointer evaluating interface {}.enabled

The public Helm chart is currently not under active development, and no new features are planned. However, Cockroach Labs remains committed to fully supporting the Helm chart by addressing defects, providing security patches, and addressing breaking changes due to deprecations in Kubernetes APIs.

A deprecation notice for the public Helm chart will be provided to customers a minimum of 6 months in advance of actual deprecation.

Network

Service Name Indication (SNI) is an extension to the TLS protocol which allows a client to indicate which hostname it is attempting to connect to at the start of the TCP handshake process. The server can present multiple certificates on the same IP address and TCP port number, and one server can serve multiple secure websites or API services even if they use different certificates.

Due to its order of operations, the PostgreSQL wire protocol’s implementation of TLS is not compatible with SNI-based routing in the Kubernetes ingress controller. Instead, use a TCP load balancer for CockroachDB that is not shared with other services.

Resources

When starting Kubernetes, select machines with at least 4 vCPUs and 16 GiB of memory, and provision at least 2 vCPUs and 8 Gi of memory to CockroachDB per pod. These minimum settings are used by default in this deployment guide, and are appropriate for testing purposes only. On a production deployment, you should adjust the resource settings for your workload. For details, see .

Storage

Kubernetes deployments use external persistent volumes that are often replicated by the provider. CockroachDB replicates data automatically, and this redundant layer of replication can impact performance. Using local volumes may improve performance.

Exposing DNS servers

Instead of using this approach, you can now enable global access, which exposes the Kubernetes cluster’s DNS servers via a load-balanced IP address that is visible to the public internet. With this this approach, noone of the services in your Kubernetes cluster will be accessible publicly, but their names could leak out to a motivated attacker.

Step 1. Start Kubernetes clusters

Our multi-region deployment approach relies on pod IP addresses being routable across three distinct Kubernetes clusters and regions. Both the hosted Google Kubernetes Engine (GKE) and Amazon Elastic Kubernetes Service (EKS) satisfy this requirement.

1. Complete the Before You Begin steps described in the Google Kubernetes Engine Quickstart documentation.

   This includes installing gcloud, which is used to create and delete Kubernetes Engine clusters, and kubectl, which is the command-line tool used to manage Kubernetes from your workstation.

2. From your local workstation, start the first Kubernetes cluster, specifying the zone it should run in:

   $ gcloud container clusters create cockroachdb1 --zone=<gce-zone
   
   Creating cluster cockroachdb1...done.
   

   This creates GKE instances in the zone specified and joins them into a single Kubernetes cluster named cockroachdb1.

The process can take a few minutes, so do not move on to the next step until you see a Creating cluster cockroachdb1...done message and details about your cluster.

1. Start the second Kubernetes cluster, specifying the zone it should run in:

   $ gcloud container clusters create cockroachdb2 --zone=<gce-zone
   
   Creating cluster cockroachdb2...done.
   

2. Start the third Kubernetes cluster, specifying the zone it should run in:

   $ gcloud container clusters create cockroachdb3 --zone=<gce-zone
   
   Creating cluster cockroachdb3...done.
   

3. Get the kubectl “contexts” for your clusters:

   $ kubectl config get-contexts
   
   CURRENT   NAME                                                  CLUSTER                                               AUTHINFO                                              NAMESPACE
   *         gke_cockroach-shared_us-east1-b_cockroachdb1          gke_cockroach-shared_us-east1-b_cockroachdb1          gke_cockroach-shared_us-east1-b_cockroachdb1
             gke_cockroach-shared_us-west1-a_cockroachdb2          gke_cockroach-shared_us-west1-a_cockroachdb2          gke_cockroach-shared_us-west1-a_cockroachdb2
             gke_cockroach-shared_us-central1-a_cockroachdb3       gke_cockroach-shared_us-central1-a_cockroachdb3       gke_cockroach-shared_us-central1-a_cockroachdb3
   

kubectl commands are run against the CURRENT context by default. You can change the current context with this command:

kubectl config use-context <context-name>

When sending commands to another context, you need to use the --context flag to specify the context. For clarity, every kubectl command in this tutorial uses the --context flag to indicate the proper context.

1. Get the email address associated with your Google Cloud account:

   $ gcloud info | grep Account
   
   Account: [[e-mail]]
   

This command returns your email address in all lowercase. However, in the next step, you must enter the address using the accurate capitalization. For example, if your address is [e-mail], you must use [e-mail] and not [e-mail].

1. For each Kubernetes cluster, create the RBAC roles CockroachDB needs for running on GKE, using the email address and relevant “context” name from the previous steps:

   $ kubectl create clusterrolebinding $USER-cluster-admin-binding --clusterrole=cluster-admin --user=<[e-mail] --context <cluster-context-1>
   
   $ kubectl create clusterrolebinding $USER-cluster-admin-binding --clusterrole=cluster-admin --user=<[e-mail] --context <cluster-context-2>
   
   $ kubectl create clusterrolebinding $USER-cluster-admin-binding --clusterrole=cluster-admin --user=<[e-mail] --context <cluster-context-3>
   

Step 2. Start CockroachDB

1. Create a directory and download the required script and configuration files into it:

   $ mkdir multiregion
   
   $ cd multiregion
   
   $ curl -OOOOOOOOO \
   https://www.cockroachlabs.com/docs/files/cockroach/cloud/kubernetes/multiregion/README.md https://www.cockroachlabs.com/docs/files/cockroach/cloud/kubernetes/multiregion/client-secure.yaml https://www.cockroachlabs.com/docs/files/cockroach/cloud/kubernetes/multiregion/cluster-init-secure.yaml https://www.cockroachlabs.com/docs/files/cockroach/cloud/kubernetes/multiregion/cockroachdb-statefulset-secure.yaml https://www.cockroachlabs.com/docs/files/cockroach/cloud/kubernetes/multiregion/dns-lb.yaml https://www.cockroachlabs.com/docs/files/cockroach/cloud/kubernetes/multiregion/example-app-secure.yaml https://www.cockroachlabs.com/docs/files/cockroach/cloud/kubernetes/multiregion/external-name-svc.yaml https://www.cockroachlabs.com/docs/files/cockroach/cloud/kubernetes/multiregion/setup.py https://www.cockroachlabs.com/docs/files/cockroach/cloud/kubernetes/multiregion/teardown.py
   

2. Retrieve the kubectl “contexts” for your clusters:

   $ kubectl config get-contexts
   

   At the top of the setup.py script, fill in the contexts map with the zones of your clusters and their “context” names, e.g.:

   contexts = {
       'us-east1-b': 'gke_cockroach-shared_us-east1-b_cockroachdb1',
       'us-west1-a': 'gke_cockroach-shared_us-west1-a_cockroachdb2',
       'us-central1-a': 'gke_cockroach-shared_us-central1-a_cockroachdb3',
   }
   

3. In the setup.py script, fill in the regions map with the zones and corresponding regions of your clusters, for example:

   $ regions = {
       'us-east1-b': 'us-east1',
       'us-west1-a': 'us-west1',
       'us-central1-a': 'us-central1',
   }
   

   Setting regions is optional, but recommended, because it improves CockroachDB’s ability to diversify data placement if you use more than one zone in the same region. If you aren’t specifying regions, just leave the map empty.

4. If you haven’t already, . The cockroach binary will be used to generate certificates.

   If the cockroach binary is not on your PATH, in the setup.py script, set the cockroach_path variable to the path to the binary.

5. Optionally, to optimize your deployment for better performance, review and make the desired modifications to the cockroachdb-statefulset-secure.yaml file.

6. Run the setup.py script:

   $ python setup.py
   

   As the script creates various resources and creates and initializes the CockroachDB cluster, you’ll see a lot of output, eventually ending with job "cluster-init-secure" created.

7. Confirm that the CockroachDB pods in each cluster say 1/1 in the READY column, indicating that they’ve successfully joined the cluster:

   $ kubectl get pods --selector app=cockroachdb --all-namespaces --context <cluster-context-1>
   
   NAMESPACE    NAME            READY     STATUS    RESTARTS   AGE
   us-east1-b   cockroachdb-0   1/1       Running   0          14m
   us-east1-b   cockroachdb-1   1/1       Running   0          14m
   us-east1-b   cockroachdb-2   1/1       Running   0          14m
   
   $ kubectl get pods --selector app=cockroachdb --all-namespaces --context <cluster-context-2>
   
   NAMESPACE       NAME            READY     STATUS    RESTARTS   AGE
   us-central1-a   cockroachdb-0   1/1       Running   0          14m
   us-central1-a   cockroachdb-1   1/1       Running   0          14m
   us-central1-a   cockroachdb-2   1/1       Running   0          14m
   
   $ kubectl get pods --selector app=cockroachdb --all-namespaces --context <cluster-context-3>
   
   NAMESPACE    NAME            READY     STATUS    RESTARTS   AGE
   us-west1-a   cockroachdb-0   1/1       Running   0          14m
   us-west1-a   cockroachdb-1   1/1       Running   0          14m
   us-west1-a   cockroachdb-2   1/1       Running   0          14m
   

   If you notice that only one of the Kubernetes clusters’ pods are marked as READY, you likely also need to configure a network firewall rule that will allow the pods in the different clusters to talk to each other. You can run the following command to create a firewall rule allowing traffic on port 26257 (the port used by CockroachDB for inter-node traffic) within your private GCE network. It will not allow any traffic in from outside your private network:

   $ gcloud compute firewall-rules create allow-cockroach-internal --allow=tcp:26257 --source-ranges=10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
   
   Creating firewall...done.
   NAME                      NETWORK  DIRECTION  PRIORITY  ALLOW      DENY
   allow-cockroach-internal  default  INGRESS    1000      tcp:26257
   

In each Kubernetes cluster, the StatefulSet configuration sets all CockroachDB nodes to write to stderr, so if you ever need access to a pod/node’s logs to troubleshoot, use kubectl logs <podname> --context <cluster-context> --namespace <cluster-namespace> rather than checking the log on the persistent volume.

Step 3. Use the built-in SQL client

Use the client-secure.yaml file to launch a pod and keep it running indefinitely, specifying the context of the Kubernetes cluster to run it in:

$ kubectl create -f client-secure.yaml --context <cluster-context>

pod "cockroachdb-client-secure" created

The pod uses the root client certificate created earlier by the setup.py script. Note that this will work from any of the three Kubernetes clusters as long as you use the correct namespace and context combination.

Step 4. Access the DB Console

To access the cluster’s :

1. On secure clusters, can only be accessed by admin users.

   Get a shell into the pod with the cockroach binary created earlier and start the CockroachDB :

   $ kubectl exec -it cockroachdb-client-secure --context <cluster-context> --namespace <cluster-namespace> -- ./cockroach sql --certs-dir=/cockroach-certs --host=cockroachdb-public
   

2. Assign roach to the admin role (you only need to do this once):

   > GRANT admin TO roach;
   

3. Exit the SQL shell and pod:

   > \q
   

4. Port-forward from your local machine to a pod in one of your Kubernetes clusters:

   $ kubectl port-forward cockroachdb-0 8080 --context <cluster-context> --namespace <cluster-namespace>
   
   Forwarding from 127.0.0.1:8080 -> 8080
   

The port-forward command must be run on the same machine as the web browser in which you want to view the DB Console. If you have been running these commands from a cloud instance or other non-local shell, you will not be able to view the UI without configuring kubectl locally and running the above port-forward command on your local machine.

1. Go to https://localhost:8080 and log in with the username and password created in the Use the built-in SQL client step.

2. In the UI, check the to verify that all nodes are running, open the to verify that bank is listed, and open the to see the performance of your CockroachDB cluster across 3 regions.

Step 5. Simulate datacenter failure

One of the major benefits of running a multi-region CockroachDB cluster is that an entire datacenter or region can go down without affecting the availability of the cluster as a whole.

To see this in action:

1. Scale down one of the StatefulSets to zero pods, specifying the namespace and context of the Kubernetes cluster where it’s running:

   $ kubectl scale statefulset cockroachdb --replicas=0 --context <cluster-context> --namespace <cluster-namespace>
   
   statefulset "cockroachdb" scaled
   

2. In the DB Console, the Cluster Overview will soon show the three nodes from that region as Suspect. If you wait for 5 minutes or more, they will be listed as Dead. Note that even though there are three dead nodes, the other nodes are all healthy, and any clients using the database in the other regions will continue to work just fine.

3. When you’re done verifying that the cluster still fully functions with one of the regions down, you can bring the region back up by running:

   $ kubectl scale statefulset cockroachdb --replicas=3 --context <cluster-context> --namespace <cluster-namespace>
   
   statefulset "cockroachdb" scaled
   

Step 6. Maintain the cluster

• Scale the cluster
• Upgrade the cluster
• Stop the cluster

Scale the cluster

Each of your Kubernetes clusters contains 3 instances that can run CockroachDB pods. It’s easy to scale a cluster to run more pods. To ensure that you do not have two CockroachDB pods on the same instance (as recommended in our ), you need to first add a new instance and then edit your StatefulSet configuration to add another pod.

1. Resize your Kubernetes cluster.

2. Use the kubectl scale command to add a pod to the StatefulSet in the Kubernetes cluster where you want to add a CockroachDB node:

   $ kubectl scale statefulset cockroachdb --replicas=4 --context <cluster-context> --namespace <cluster-namespace>
   
   statefulset "cockroachdb" scaled
   

3. Verify that a fourth pod was added successfully:

   $ kubectl get pods --context <cluster-context> --namespace <cluster-namespace>
   
   NAME                        READY     STATUS    RESTARTS   AGE
   cockroachdb-0               1/1       Running   0          1h
   cockroachdb-1               1/1       Running   0          1h
   cockroachdb-2               1/1       Running   0          7m
   cockroachdb-3               1/1       Running   0          44s
   cockroachdb-client-secure   1/1       Running   0          26m
   

Upgrade the cluster

We strongly recommend that you regularly upgrade your CockroachDB version in order to pick up bug fixes, performance improvements, and new features.

The upgrade process on Kubernetes is a staged update in which the Docker image is applied to the pods one at a time, with each pod being stopped and restarted in turn. This is to ensure that the cluster remains available during the upgrade.

1. Verify that you can upgrade.

   To upgrade to a new major version, you must first be on a production release of the previous version. The release does not need to be the latest production release of the previous version, but it must be a production and not a testing release (alpha/beta).

   Therefore, in order to upgrade to v26.3, you must be on a production release of .

   1. If you are upgrading to v26.3 from a production release earlier than v26.2, or from a testing release (alpha/beta), first upgrade to a production release of v26.2. Be sure to complete all the steps.

   2. Then return to this page and perform a second upgrade to v26.3.

   3. If you are upgrading from any production release of , or from any earlier v26.3 release, you do not have to go through intermediate releases; continue to step 2.

2. Verify the overall health of your cluster using the . On the Overview:

   • Under Node Status, make sure all nodes that should be live are listed as such. If any nodes are unexpectedly listed as suspect or dead, identify why the nodes are offline and either restart them or decommission them before beginning your upgrade. If there are dead and non-decommissioned nodes in your cluster, it will not be possible to finalize the upgrade (either automatically or manually).
   • Under Replication Status, make sure there are 0 under-replicated and unavailable ranges. Otherwise, performing a rolling upgrade increases the risk that ranges will lose a majority of their replicas and cause cluster unavailability. Therefore, it’s important to identify and resolve the cause of range under-replication and/or unavailability before beginning your upgrade.
   • In the Node List:
     • Make sure all nodes are on the same version. If any nodes are behind, upgrade them to the cluster’s current version first, and then start this process over.
     • Make sure capacity and memory usage are reasonable for each node. Nodes must be able to tolerate some increase in case the new version uses more resources for your workload. Also go to Metrics > Dashboard: Hardware and make sure CPU percent is reasonable across the cluster. If there’s not enough headroom on any of these metrics, consider adding nodes to your cluster before beginning your upgrade.

3. Review the backward-incompatible changes in v26.3 and deprecated features. If any affect your deployment, make the necessary changes before starting the rolling upgrade to v26.3.

4. Decide how the upgrade will be finalized.

   By default, after all nodes are running the new version, the upgrade process will be auto-finalized. This will enable certain features and performance improvements introduced in v26.3. After finalization, however, it will no longer be possible to perform a downgrade to . In the event of a catastrophic failure or corruption, the only option is to start a new cluster using the old binary and then restore from a created prior to the upgrade. For this reason, we recommend disabling auto-finalization so you can monitor the stability and performance of the upgraded cluster before finalizing the upgrade, but note that you will need to follow all of the subsequent directions, including the manual finalization in a later step.

Finalization only applies when performing a major version upgrade (for example, from .x to v26.3). Patch version upgrades (for example, within the v26.3.x series) can always be downgraded.

1. Get a shell into the pod with the cockroach binary created earlier and start the CockroachDB :

   $ kubectl exec -it cockroachdb-client-secure --context <cluster-context> --namespace <cluster-namespace> -- ./cockroach sql --certs-dir=/cockroach-certs --host=cockroachdb-public
   

2. Set the cluster.preserve_downgrade_option :

   > SET CLUSTER SETTING cluster.preserve_downgrade_option = '21.1';
   

3. For each Kubernetes cluster, kick off the upgrade process by changing the desired Docker image. To do so, pick the version that you want to upgrade to, then run the following command, replacing “VERSION” with your desired new version and specifying the relevant namespace and “context” name for the Kubernetes cluster:

   $ kubectl patch statefulset cockroachdb --namespace <namespace-of-kubernetes-cluster1> --context <cluster-context-1> --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"cockroachdb/cockroach:VERSION"}]'
   
   $ kubectl patch statefulset cockroachdb --namespace <namespace-of-kubernetes-cluster2> --context <cluster-context-2> --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"cockroachdb/cockroach:VERSION"}]'
   
   $ kubectl patch statefulset cockroachdb --namespace <namespace-of-kubernetes-cluster3> --context <cluster-context-3> --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"cockroachdb/cockroach:VERSION"}]'
   

4. If you then check the status of the pods in each Kubernetes cluster, you should see one of them being restarted:

   $ kubectl get pods --selector app=cockroachdb --all-namespaces --context <cluster-context-1>
   
   $ kubectl get pods --selector app=cockroachdb --all-namespaces --context <cluster-context-2>
   
   $ kubectl get pods --selector app=cockroachdb --all-namespaces --context <cluster-context-3>
   

   This will continue until all of the pods have restarted and are running the new image.

5. If you disabled auto-finalization earlier, monitor the stability and performance of your cluster until you are comfortable with the upgrade (generally at least a day).

   If you decide to roll back the upgrade, repeat the rolling restart procedure with the old binary.

This is only possible when performing a major version upgrade (for example, from .x to v26.3). Patch version upgrades (for example, within the v26.3.x series) are auto-finalized.

To finalize the upgrade, re-enable auto-finalization:

1. Get a shell into the pod with the cockroach binary created earlier and start the CockroachDB :

   $ kubectl exec -it cockroachdb-client-secure --context <cluster-context> --namespace <cluster-namespace> -- ./cockroach sql --certs-dir=/cockroach-certs --host=cockroachdb-public
   

2. Re-enable auto-finalization:

   > RESET CLUSTER SETTING cluster.preserve_downgrade_option;
   

Stop the cluster

1. To delete all of the resources created in your clusters, copy the contexts map from setup.py into teardown.py, and then run teardown.py:

   $ python teardown.py
   
   namespace "us-east1-b" deleted
   service "kube-dns-lb" deleted
   configmap "kube-dns" deleted
   pod "kube-dns-5dcfcbf5fb-l4xwt" deleted
   pod "kube-dns-5dcfcbf5fb-tddp2" deleted
   namespace "us-west1-a" deleted
   service "kube-dns-lb" deleted
   configmap "kube-dns" deleted
   pod "kube-dns-5dcfcbf5fb-8csc9" deleted
   pod "kube-dns-5dcfcbf5fb-zlzn7" deleted
   namespace "us-central1-a" deleted
   service "kube-dns-lb" deleted
   configmap "kube-dns" deleted
   pod "kube-dns-5dcfcbf5fb-6ngmw" deleted
   pod "kube-dns-5dcfcbf5fb-lcfxd" deleted
   

2. Stop each Kubernetes cluster:

   $ gcloud container clusters delete cockroachdb1 --zone=<gce-zone
   
   Deleting cluster cockroachdb1...done.
   
   $ gcloud container clusters delete cockroachdb2 --zone=<gce-zone
   
   Deleting cluster cockroachdb2...done.
   
   $ gcloud container clusters delete cockroachdb3 --zone=<gce-zone
   
   Deleting cluster cockroachdb3...done.
   

See also