getAccount(int id) {
return mapper.findAccountById(id);
}
@Override
@Transactional(propagation = Propagation.REQUIRES_NEW)
public int transferFunds(int fromId, int toId, int amount) {
return mapper.transfer(fromId, toId, amount);
}
@Override
@Transactional(propagation = Propagation.REQUIRES_NEW)
public long findCountOfAccounts() {
return mapper.findCountOfAccounts();
}
@Override
@Transactional(propagation = Propagation.REQUIRES_NEW)
public int deleteAllAccounts() {
return mapper.deleteAllAccounts();
}
}
```
Note that the public methods (i.e., the methods to be called by other classes in the project) are annotated as [`@Transactional`](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#transaction-declarative-annotations) methods. This ensures that all of the SQL statements executed in the data access layer are run within the context of a database transaction
`@Transactional` takes a number of parameters, including a `propagation` parameter that determines the transaction propagation behavior around an object (i.e., at what point in the stack a transaction starts and ends). `propagation=REQUIRES_NEW` for the methods in the service layer, meaning that a new transaction must be created each time a request is made to the service layer. With this propagation behavior, the application follows the [entity-control-boundary (ECB) pattern](https://wikipedia.org/wiki/Entity-control-boundary), as the service boundaries determine where a transaction starts and ends rather than the lower-level query methods of the [mapper interfaces](#mappers).
For more details on aspect-oriented transaction management in this application, [see below](#transaction-management).
### Models
Instances of the `Account` class, defined in `src/main/java/com/example/cockroachdemo/model/Account.java`, represent rows in the `accounts` table:
```java theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
package com.example.cockroachdemo.model;
public class Account {
private int id;
private int balance;
public int getId() {
return id;
}
public void setId(int id) {
this.id = id;
}
public int getBalance() {
return balance;
}
public void setBalance(int balance) {
this.balance = balance;
}
}
```
Instances of the `BatchResults` class, defined in `src/main/java/com/example/cockroachdemo/model/BatchResults.java`, hold metadata about a batch write operation and its results:
```java theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
package com.example.cockroachdemo.model;
public class BatchResults {
private int numberOfBatches;
private int totalRowsAffected;
public BatchResults(int numberOfBatches, int totalRowsAffected) {
this.numberOfBatches = numberOfBatches;
this.totalRowsAffected = totalRowsAffected;
}
public int getNumberOfBatches() {
return numberOfBatches;
}
public int getTotalRowsAffected() {
return totalRowsAffected;
}
}
```
### Transaction management
MyBatis-Spring supports Spring's [declarative, aspect-oriented transaction management syntax](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#transaction-declarative), including the [`@Transactional`](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#transaction-declarative-annotations) annotation and [AspectJ's AOP annotations](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#transaction-declarative-aspectj).
Transactions may require retries if they experience deadlock or transaction contention that cannot be resolved without allowing serialization anomalies. To handle transactions that are aborted due to transient serialization errors, we highly recommend writing client-side transaction retry logic into applications written on CockroachDB. In this application, transaction retry logic is written into the methods of the `RetryableTransactionAspect` class, defined in `src/main/java/com/example/cockroachdemo/RetryableTransactionAspect.java`:
```java theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
package com.example.cockroachdemo;
import java.lang.reflect.UndeclaredThrowableException;
import java.util.concurrent.atomic.AtomicLong;
import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.annotation.Around;
import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Pointcut;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.core.Ordered;
import org.springframework.core.annotation.Order;
import org.springframework.dao.ConcurrencyFailureException;
import org.springframework.dao.TransientDataAccessException;
import org.springframework.stereotype.Component;
import org.springframework.transaction.TransactionSystemException;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.transaction.support.TransactionSynchronizationManager;
import org.springframework.util.Assert;
/**
* Aspect with an around advice that intercepts and retries transient concurrency exceptions.
* Methods matching the pointcut expression (annotated with @Transactional) are retried.
*
* This advice needs to run in a non-transactional context, which is before the underlying
* transaction advisor (@Order ensures that).
*/
@Component
@Aspect
// Before TX advisor
@Order(Ordered.LOWEST_PRECEDENCE - 1)
public class RetryableTransactionAspect {
protected final Logger logger = LoggerFactory.getLogger(getClass());
@Pointcut("@annotation(transactional)")
public void anyTransactionBoundaryOperation(Transactional transactional) {
}
@Around(value = "anyTransactionBoundaryOperation(transactional)",
argNames = "pjp,transactional")
public Object retryableOperation(ProceedingJoinPoint pjp, Transactional transactional)
throws Throwable {
final int totalRetries = 30;
int numAttempts = 0;
AtomicLong backoffMillis = new AtomicLong(150);
Assert.isTrue(!TransactionSynchronizationManager.isActualTransactionActive(), "TX active");
do {
try {
numAttempts++;
return pjp.proceed();
} catch (TransientDataAccessException | TransactionSystemException ex) {
handleTransientException(ex, numAttempts, totalRetries, pjp, backoffMillis);
} catch (UndeclaredThrowableException ex) {
Throwable t = ex.getUndeclaredThrowable();
if (t instanceof TransientDataAccessException) {
handleTransientException(t, numAttempts, totalRetries, pjp, backoffMillis);
} else {
throw ex;
}
}
} while (numAttempts < totalRetries);
throw new ConcurrencyFailureException("Too many transient errors (" + numAttempts + ") for method ["
+ pjp.getSignature().toLongString() + "]. Giving up!");
}
private void handleTransientException(Throwable ex, int numAttempts, int totalAttempts,
ProceedingJoinPoint pjp, AtomicLong backoffMillis) {
if (logger.isWarnEnabled()) {
logger.warn("Transient data access exception (" + numAttempts + " of max " + totalAttempts + ") "
+ "detected (retry in " + backoffMillis + " ms) "
+ "in method '" + pjp.getSignature().getDeclaringTypeName() + "." + pjp.getSignature().getName()
+ "': " + ex.getMessage());
}
if (backoffMillis.get() >= 0) {
try {
Thread.sleep(backoffMillis.get());
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
backoffMillis.set(Math.min((long) (backoffMillis.get() * 1.5), 1500));
}
}
}
```
The [`@Aspect` annotation](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-at-aspectj) declares `RetryableTransactionAspect` an [aspect](https://wikipedia.org/wiki/Aspect_\(computer_programming\)), with [pointcut](https://wikipedia.org/wiki/Pointcut) and [advice](https://wikipedia.org/wiki/Advice_\(programming\)) methods.
#### Transactional pointcut
The [`@Pointcut` annotation](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-pointcuts) declares the `anyTransactionBoundaryOperation` method the pointcut for determining when to execute the aspect's advice. The `@annotation` [designator](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-pointcuts-designators) passed to the `@Pointcut` annotation limits the matches (i.e., [join points](https://wikipedia.org/wiki/Join_point)) to method calls with a specific annotation, in this case, `@Transactional`.
#### Transaction retry advice
`retryableOperation` handles the application retry logic, with [exponential backoff](https://wikipedia.org/wiki/Exponential_backoff), as the advice to execute at an `anyTransactionBoundaryOperation(transactional)` join point. Spring supports [several different annotations to declare advice](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-advice). The [`@Around` annotation](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-ataspectj-around-advice) allows an advice method to work before and after the join point. It also gives the advice method control over executing any other matching advisors.
`retryableOperation` first verifies that there is no active transaction. It then increments the retry count and attempts to proceed to the next advice method with the `ProceedingJoinPoint.proceed()` method. If the underlying data access layer method (i.e., the mapper interface method annotated with `@Transactional`) succeeds, the results are returned and the application flow continues. If the method fails, then the transaction is retried. The time between each retry grows with each retry until the maximum number of retries is reached.
#### Advice ordering
Spring automatically adds [transaction management advice](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#tx-decl-explained) to all methods annotated with `@Transactional`. Because the pointcut for `RetryableTransactionAspect` also matches methods annotated with `@Transactional`, there will always be two advisors that match the same pointcut. When multiple advisors match at the same pointcut, an `@Order` annotation on an advisor's aspect can specify the order in which the advice should be evaluated.
To control when and how often a transaction is retried, the transaction retry advice must be executed outside the context of a transaction (i.e., it must be evaluated before the primary transaction management advisor). By default, the primary transaction management advisor is given the lowest level of precedence. The `@Order` annotation on `RetryableTransactionAspect` is passed `Ordered.LOWEST_PRECEDENCE-1`, which places this aspect's advice at a level of precedence above the primary transaction advisor, which results in the retry logic being evaluated before the transaction management advisor.
For more details about advice ordering in Spring, see [Advice Ordering](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-ataspectj-advice-ordering) on the Spring documentation site.
## See also
Spring documentation:
* [Spring Boot website](https://spring.io/projects/spring-boot)
* [Spring Framework Overview](https://docs.spring.io/spring/docs/current/spring-framework-reference/overview.html#overview)
* [Spring Core documentation](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#spring-core)
* [MyBatis documentation](https://mybatis.org/mybatis-3/)
* [MyBatis Spring integration](https://mybatis.org/spring/)
CockroachDB documentation:
* Learn CockroachDB SQL
* Client Connection Parameters
* CockroachDB Developer Guide
* Transactions
# Build a TypeScript App with CockroachDB and TypeORM
Source: https://docs.cockroachlabs.com/docs/v26.3/build-a-typescript-app-with-cockroachdb
This tutorial shows you how run a simple application built with [TypeORM](https://typeorm.io/#/).
## Step 1. Start CockroachDB
### Choose your installation method
You can create a CockroachDB Basic cluster using either the CockroachDB Cloud Console, a web-based graphical user interface (GUI) tool, or `ccloud`, a command-line interface (CLI) tool.
### Create a free cluster
Organizations without billing information on file can only create one CockroachDB Basic cluster.
1. If you haven't already, [sign up for a CockroachDB Cloud account](https://cockroachlabs.cloud/signup?referralId=docs_typescript).
2. [Log in](https://cockroachlabs.cloud/) to your CockroachDB Cloud account.
3. On the **Clusters** page, click **Create cluster**.
4. On the **Select a plan** page, select **Basic**.
5. On the **Cloud & Regions** page, select a cloud provider (GCP or AWS) in the **Cloud provider** section.
6. In the **Regions** section, select a region for the cluster. Refer to CockroachDB Cloud Regions for the regions where CockroachDB Basic clusters can be deployed. To create a multi-region cluster, click **Add region** and select additional regions.
7. Click **Next: Capacity**.
8. On the **Capacity** page, select **Start for free**. Click **Next: Finalize**.
9. On the **Finalize** page, click **Create cluster**.
Your cluster will be created in a few seconds and the **Create SQL user** dialog will display.
### Create a SQL user
The **Create SQL user** dialog allows you to create a new SQL user and password.
1. Enter a username in the **SQL user** field or use the one provided by default.
2. Click **Generate & save password**.
3. Copy the generated password and save it in a secure location.
4. Click **Next**.
Currently, all new SQL users are created with admin privileges. For more information and to change the default settings, see Manage SQL users on a cluster.
### Get the connection string
The **Connect to cluster** dialog shows information about how to connect to your cluster.
1. Select **General connection string** from the **Select option** dropdown.
2. Open the **General connection string** section, then copy the connection string provided and save it in a secure location.
The sample application used in this tutorial uses system CA certificates for server certificate verification, so you can skip the **Download CA Cert** instructions.
The connection string is pre-populated with your username, password, cluster name, and other details. Your password, in particular, will be provided *only once*. Save it in a secure place (Cockroach Labs recommends a password manager) to connect to your cluster in the future. If you forget your password, you can reset it by going to the **SQL Users** page for the cluster, found at `https://cockroachlabs.cloud/cluster//users`.
Follow these steps to create a CockroachDB Basic cluster using the `ccloud` CLI tool.
The `ccloud` CLI tool is in Preview.
### Install `ccloud`
You can install `ccloud` using either Homebrew or by downloading the binary.
#### Use Homebrew
1. [Install Homebrew](http://brew.sh/).
2. Install using the `ccloud` tap:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
brew install cockroachdb/tap/ccloud
```
#### Download the binary
In a terminal, enter the following command to download and extract the `ccloud` binary and add it to your `PATH`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl https://binaries.cockroachdb.com/ccloud/ccloud_darwin-amd64_.tar.gz | tar -xJ && cp -i ccloud /usr/local/bin/
```
Use the ARM 64 binary if you have an M-series Mac:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl https://binaries.cockroachdb.com/ccloud/ccloud_darwin-arm64_.tar.gz | tar -xJ && cp -i ccloud /usr/local/bin/
```
In a terminal, enter the following command to download and extract the `ccloud` binary and add it to your `PATH`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl https://binaries.cockroachdb.com/ccloud/ccloud_linux-amd64_.tar.gz | tar -xz && cp -i ccloud /usr/local/bin/
```
In a PowerShell window, enter the following command to download and extract the `ccloud` binary and add it to your `PATH`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ErrorActionPreference = "Stop"; [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; $ProgressPreference = 'SilentlyContinue'; $null = New-Item -Type Directory -Force $env:appdata/ccloud; Invoke-WebRequest -Uri https://binaries.cockroachdb.com/ccloud/ccloud_windows-amd64_.zip -OutFile ccloud.zip; Expand-Archive -Force -Path ccloud.zip; Copy-Item -Force ccloud/ccloud.exe -Destination $env:appdata/ccloud; $Env:PATH += ";$env:appdata/ccloud"; # We recommend adding ";$env:appdata/ccloud" to the Path variable for your system environment. See https://docs.microsoft.com/powershell/module/microsoft.powershell.core/about/about_environment_variables#saving-changes-to-environment-variables for more information.
```
### Run `ccloud quickstart` to create a new cluster, create a SQL user, and retrieve the connection string.
The easiest way of getting started with CockroachDB Cloud is to use `ccloud quickstart`. The `ccloud quickstart` command guides you through logging in to CockroachDB Cloud, creating a new CockroachDB Basic cluster, and connecting to the new cluster. Run `ccloud quickstart` and follow the instructions:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
ccloud quickstart
```
The `ccloud quickstart` command will open a browser window to log you in to CockroachDB Cloud. If you are new to CockroachDB Cloud, you can register using one of the single sign-on (SSO) options, or create a new account using an email address.
The `ccloud quickstart` command will prompt you for the cluster name, cloud provider, and cloud provider region, then ask if you want to connect to the cluster. Each prompt has default values that you can select, or change if you want a different option.
Select **General connection string**, then copy the connection string displayed and save it in a secure location. The connection string is the line starting `postgresql://`.
```
? How would you like to connect? General connection string
Retrieving cluster info: succeeded
Downloading cluster cert to /Users/maxroach/.postgresql/root.crt: succeeded
postgresql://maxroach:ThisIsNotAGoodPassword@blue-dog-147.6wr.cockroachlabs.cloud:26257/defaultdb?sslmode=verify-full&sslrootcert=%2FUsers%2Fmaxroach%2F.postgresql%2Froot.crt
```
1. If you haven't already, download the CockroachDB binary.
2. Run the `cockroach start-single-node` command:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start-single-node --advertise-addr 'localhost' --insecure
```
This starts an insecure, single-node cluster.
3. Take note of the following connection information in the SQL shell welcome text:
```
CockroachDB node starting at 2021-08-30 17:25:30.06524 +0000 UTC (took 4.3s)
build: CCL v21.1.6 @ 2021/07/20 15:33:43 (go1.15.11)
webui: http://localhost:8080
sql: postgresql://root@localhost:26257?sslmode=disable
```
You'll use the `sql` connection string to connect to the cluster later in this tutorial.
The `--insecure` flag used in this tutorial is intended for non-production testing only. To run CockroachDB in production, use a secure cluster instead.
## Step 2. Get the code
1. Clone [the code's GitHub repository](https://github.com/cockroachlabs/example-app-typescript-typeorm):
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ git clone git@github.com:cockroachlabs/example-app-typescript-typeorm.git
```
2. Navigate to the repo directory and install the application dependencies:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cd example-app-typescript-typeorm
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ npm install
```
## Step 3. Configure your CockroachDB connection
1. Open the `datasource.ts` file, and comment out the `ssl: true`, `extra` and `options` configuration properties.
2. In the `datasource.ts` file, uncomment `ssl: { rejectUnauthorized: false }`.
Only use `ssl: { rejectUnauthorized: false }` in development, for insecure connections.
The `DataSource` configuration should look similar to the following:
```ts theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export const AppDataSource = new DataSource({
type: "cockroachdb",
url: process.env.DATABASE_URL,
ssl: { rejectUnauthorized: false }, // For insecure connections only
synchronize: true,
logging: false,
entities: ["src/entity/**/*.ts"],
migrations: ["src/migration/**/*.ts"],
subscribers: ["src/subscriber/**/*.ts"],
})
```
1. Set the `DATABASE_URL` environment variable to the connection string provided in the `cockroach` welcome text:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="postgresql://root@localhost:26257/defaultdb?sslmode=disable"
```
2. Set the `DATABASE_URL` environment variable to a CockroachDB connection string compatible with TypeORM.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="CockroachDB Cloud MCP server.
* Cluster-level read and write access for self-hosted deployments via the CockroachDB MCP Server.
* Operational workflows encoded as Agent Skills for CockroachDB.
* Command-line automation with the `ccloud` CLI, designed to be compatible with AI agents.
* Access to the Cockroach Labs public documentation via the CockroachDB Docs MCP server.
The following sections provide an overview of these tools.
### CockroachDB MCP servers
CockroachDB provides Model Context Protocol (MCP) servers that expose a set of tools for inspecting and querying your clusters from your AI tools. These tools let your AI assistants list databases and tables, describe schemas and indexes, inspect cluster health and running queries, and run read-only SQL and `EXPLAIN` statements. When explicitly enabled, they can also create databases, create tables, and insert rows.
This allows you to manage and modify a CockroachDB cluster using natural language prompts. For example, you could use the following prompt to instruct your AI tool to interface with the cluster:
```
List all of the tables in the demo_db database.
```
An MCP server is available for both deployment types: CockroachDB Cloud clusters use the managed CockroachDB Cloud MCP server, and self-hosted clusters use the CockroachDB MCP Server, which you run yourself.
### Agent Skills for CockroachDB
Agent Skills for CockroachDB are small, structured capabilities that encode CockroachDB operational expertise in a machine-executable format. These skills live in the public [cockroachdb-skills](https://github.com/cockroachlabs/cockroachdb-skills) repository and follow the [Agent Skills Specification](https://agentskills.io/specification), with defined inputs, outputs, and safety guardrails.
Each skill focuses on a specific task, such as auditing user privileges, triaging live SQL activity, validating production readiness, or checking backup and disaster recovery posture. Skills are organized into domains like onboarding and migrations, application development, performance and scaling, operations and lifecycle, resilience and disaster recovery, observability and diagnostics, security and governance, integrations, and cost management.
Your AI tools can consume these skills directly, so you can reuse the same operational workflows across different toolchains.
### `ccloud` command-line interface
The `ccloud` CLI is the command-line interface for CockroachDB Cloud. You can use the `ccloud` CLI to create clusters, manage networking (for example, IP allowlists), create SQL users, retrieve connection information, and more.
Because `ccloud` is text-based and follows a stable command structure, it is well-suited for AI tools and automations. An AI assistant can generate or run `ccloud` commands to set up clusters, rotate credentials, or retrieve connection URLs, while you keep access mediated through the CLI and existing Cloud authentication.
### CockroachDB Docs MCP server
The CockroachDB Docs MCP server exposes the published CockroachDB documentation to your MCP-compatible tools over HTTP. After you add the server configuration to your client, your AI assistant can answer questions using the official documentation without leaving your editor. For example, you could use your AI tool to ask the following:
```
How do table statistics get refreshed in CockroachDB?
```
An AI tool using the documentation MCP server can provide a response to this prompt informed by documentation pages such as Cost-Based Optimizer and `CREATE STATISTICS`.
This integration only provides access to CockroachDB product documentation. It does not connect to your clusters or data. It is intended to give your AI tools reliable product information while you develop and operate CockroachDB and CockroachDB Cloud.
## CockroachDB as a data store for AI applications
CockroachDB provides the database features needed to store and query AI-related data, including vector embeddings and agent state, with the same transactional guarantees as your other workloads.
### Vector search and RAG
AI applications often represent text, images, and other content as vector embeddings. These are numerical representations that capture semantic meaning. To find relevant information, AI applications need to search for vectors that are similar to a query vector, typically using distance metrics. This similarity search is fundamental to retrieval-augmented generation (RAG), semantic search, and recommendation systems.
CockroachDB has a `VECTOR` data type for storing fixed-length floating-point embeddings and supports similarity operators such as L2 distance (`<->`), inner product (`<#>`), and cosine distance (`<=>`). You can index vectors using `CREATE VECTOR INDEX` and combine them with other indexed columns.
You can store vector embeddings, relational data, and JSON metadata in the same table and query them together. For example:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
SELECT id, name
FROM items
WHERE customer_id = 1
ORDER BY embedding <-> $query_vector
LIMIT 5;
```
RAG systems depend on fast and accurate retrieval of contextual data. CockroachDB's vector support enables you to implement RAG and semantic search patterns so that you can use CockroachDB as the data store for AI applications. Vector indexes enable efficient similarity searches over large datasets. You can store document embeddings in a cluster alongside the source documents, metadata, and access control information associated with those documents. When a user query comes in, you can use vector similarity operators to retrieve semantically relevant documents, filter by permissions, and return both the relevant context and any associated metadata in a single transaction.
See the [RAG tutorial](https://www.cockroachlabs.com/blog/tutorial-rag-with-cockroachdb/) for a complete implementation example.
### AI agent state and workflow coordination
AI agents that perform autonomous operations require durable storage for execution state, workflow metadata, and operational history. These agents must track state transitions across multi-step processes, coordinate activities between concurrent executions, and ensure that operations can be safely retried or resumed after failures.
CockroachDB's transactional model provides a foundation for storing agent state, execution history, and coordination metadata. Serializable isolation ensures that state transitions occur correctly even when multiple agents or processes attempt concurrent updates. The database's high availability design allows agents to continue operating during node or region failures without requiring external coordination services.
### Scale, consistency, and governance
AI applications typically generate high data volumes, serve globally distributed users, and require both transactional correctness and operational durability. Conversation histories, vector embeddings, feature tables, and agent state accumulate quickly and are accessed across regions. These characteristics make AI workloads well-suited to CockroachDB's core design:
* CockroachDB scales horizontally by adding nodes to increase capacity.
* Data is automatically replicated and rebalanced across the cluster, so node failures do not require application-level failover.
* Multi-region deployments place data closer to users and can enforce data locality or residency requirements while maintaining strong consistency.
CockroachDB provides serializable transactions by default. Vector indexes participate in the same transaction and index maintenance model as other secondary indexes, so similarity search results remain aligned with the underlying data. This applies to tables storing embeddings, relational records, and agent state.
## See also
* [Tutorial: Augment your AI use case with RAG on CockroachDB](https://www.cockroachlabs.com/blog/tutorial-rag-with-cockroachdb/)
* [Real-Time Indexing for Billions of Vectors](https://www.cockroachlabs.com/blog/cspann-real-time-indexing-billions-vectors/)
* [How CockroachDB's AI Assistance Boosts Developer Productivity](https://www.cockroachlabs.com/blog/cockroachdb-ai-assistance-for-developers/)
* [Agent Development with CockroachDB using the LangChain Framework](https://www.cockroachlabs.com/blog/agent-development-cockroachdb-langchain/)
* [CockroachDB Plugin for Claude Code](https://github.com/cockroachdb/claude-plugin)
* [CockroachDB Plugin for Cursor](https://github.com/cockroachdb/cursor-plugin)
* [Zero to AI Hero](https://youtu.be/PQdeDj9QSaU)
* [Fraud Detection via Agentic AI](https://youtu.be/YzIv9yd5l_M)
* [Vector Deep Dive](https://youtu.be/rR2G35NHdZQ)
# CockroachDB MCP Server
Source: https://docs.cockroachlabs.com/docs/v26.3/cockroachdb-mcp-server
The CockroachDB MCP Server is a [Model Context Protocol (MCP)](https://modelcontextprotocol.io) server that lets your AI coding tools and AI agents explore live schemas, run queries, and write data on a CockroachDB cluster. The server runs as a local subprocess or a shared HTTP service and exposes a set of typed tools that agents invoke on your behalf. Once connected, you can interact with your cluster using natural language prompts to perform operations such as listing tables, executing `SELECT` statements, and inserting rows.
This page explains how the CockroachDB MCP Server works, and what functionality it enables. To learn how to set it up, refer to Install, Configure, and Connect to the CockroachDB MCP Server. For the full list of configuration options, refer to the CockroachDB MCP Server Reference.
For CockroachDB Cloud deployments, refer to Connect to the CockroachDB Cloud MCP Server.
## Requirements
* An existing CockroachDB cluster with a SQL user whose privileges are suitable for the operations that you want your AI agents to perform.
* Installing with `go install` or building from source requires [Go 1.26 or later](https://go.dev/dl/). Alternatively, you can install with pre-built binaries (for Linux and Windows), and a Docker image is also available.
## How it works
The server registers a set of [typed MCP tools](#tools) that AI agents invoke on your behalf. These tools trigger SQL queries against the connected cluster using a CockroachDB SQL user.
The server supports two transports:
* **stdio (default)**: Your AI tool launches the server as a local subprocess and communicates with it over standard input/output. Each user runs their own server instance, authenticated to the cluster with their own SQL credentials. This is the simplest deployment and the recommended starting point.
* **HTTP**: The server runs as a long-lived process that multiple MCP clients connect to over HTTPS with a bearer token. This is best for shared or remote deployments.
## Security
The server is configured with secure defaults and requires explicit opt-in to relax them.
* **Read-only by default.** Unless you explicitly enable write tools with the `CRDB_MCP_ENABLE_WRITE_QUERIES` environment variable, only read tools are registered, and every SQL session is additionally forced read-only with `default_transaction_read_only=true`. For per-operation control, grant the connecting SQL role only the privileges you want: for example, `SELECT`, `INSERT`, and `UPDATE` but not `DELETE` to allow writes without deletes.
* **SQL grants are the permission boundary.** Security ultimately comes from what the connecting SQL user is allowed to do in CockroachDB (via `GRANT`), not from which environment variables you set on the MCP server. The server configuration controls which tools are registered, but even with write tools enabled, the SQL user can only do what its grants allow. Least-privilege grants are also the containment for prompt injection: the server cannot distinguish a tool call you intended from one induced by malicious content that the agent has read.
* **Password authentication is rejected by default.** In stdio mode the server runs as a subprocess of the AI agent host, which can read passwords from the process environment or `~/.pgpass`. Certificate-based authentication is recommended; password-based authentication requires an explicit opt-in.
* **TLS by default, in both directions.** Connections to the cluster require `sslmode` of `require` or stronger, and HTTP mode refuses to start without a TLS certificate and key. Running without TLS—for local development or behind a TLS-terminating reverse proxy—requires explicit opt-in.
* **Background quality of service.** By default, queries run at `default_transaction_quality_of_service=background`, so agent traffic does not contend with latency-sensitive foreground workloads.
## Tools
For the full tool specifications including parameters, refer to the CockroachDB MCP Server Reference.
The server registers the following read-only tools by default:
| Tool | Description |
| - | - |
| `list_databases` | List all databases in the cluster. |
| `list_tables` | List all tables in a database. |
| `get_table_schema` | Get the schema for a table, including columns and indexes. |
| `get_cluster` | Return cluster identity and version metadata. |
| `list_sql_users` | List SQL users defined in the cluster. |
| `list_cluster_nodes` | List cluster nodes with address, liveness, and locality. |
| `show_running_queries` | List in-flight SQL statements on the cluster. |
| `select_query` | Execute a `SELECT` statement. |
| `explain_query` | Return the `EXPLAIN` plan for a statement without executing it. |
| `show_statement` | Execute a `SHOW` statement such as `SHOW SCHEMAS`, `SHOW INDEXES`, or `SHOW JOBS`. |
The following write tools are registered only when `CRDB_MCP_ENABLE_WRITE_QUERIES``=true` is set:
| Tool | Description |
| - | - |
| `create_database` | Create a database. |
| `create_table` | Execute a `CREATE TABLE` statement. |
| `insert_rows` | Execute an `INSERT` statement. |
| `update_rows` | Execute an `UPDATE` statement. A `WHERE` clause is mandatory. |
| `delete_rows` | Execute a `DELETE` statement. A `WHERE` clause is mandatory. |
## Example usage
Use natural language prompts to read from and write to the cluster. You likely do not have to invoke the preceding tools explicitly. Most AI tools should be able to decide which functions to call based on natural language prompts provided by the user.
Prompts can be simple, for example:
```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
List all of the tables in the movr database.
```
They can also be complex and conversational, for example:
```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
I need to add a reviews table to the movr database so that riders can rate their rides.
For the schema, I'm thinking we need:
- Review ID (primary key)
- Rider reference (foreign key to movr.users)
- Ride reference (foreign key to movr.rides)
- A 1–5 star rating
- Optional text feedback
- Timestamp
Let's include a secondary index on the rating column.
Show me the CREATE TABLE statement so that we can talk it through before you actually create the table.
```
## See also
* Install, Configure, and Connect to the CockroachDB MCP Server
* CockroachDB MCP Server Reference
* Connect to the CockroachDB Cloud MCP Server
* CockroachDB and AI
* Agent Skills for CockroachDB
* Docs MCP Server
# CockroachDB MCP Server Reference
Source: https://docs.cockroachlabs.com/docs/v26.3/cockroachdb-mcp-server-reference
This page lists the configuration options for the MCP Server that connects to self-hosted CockroachDB deployments. To set up the server, refer to Install, Configure, and Connect to the CockroachDB MCP Server.
## Configuration
All configuration is managed via environment variables. In stdio mode, set them in the `env` block of your MCP client configuration; in HTTP mode, export them in the server's environment.
### Authentication
Certificate-based authentication is recommended in stdio mode. The server runs as a subprocess of the AI agent host, which can read `CRDB_PWD`, a password embedded in `CRDB_DATABASE_URL`, `PGPASSWORD`, or `~/.pgpass` from the process's environment. To protect those credentials, password-based authentication is rejected by default; set [`CRDB_MCP_ALLOW_PASSWORD_AUTH`](#crdb-mcp-allow-password-auth)`=true` to opt in.
You can authenticate with individual variables (recommended) or with a full connection string. [`CRDB_DATABASE_URL`](#crdb-database-url) takes precedence when both are set.
| Variable | Description | Required | Default |
| - | - | - | - |
| `CRDB_HOST` | Hostname. For a multi-node cluster, use a load balancer address; for development, any node's address. | Yes | |
| `CRDB_PORT` | Port. | No | `26257` |
| `CRDB_USERNAME` | SQL user. | Yes | |
| `CRDB_PWD` | Password. Requires [`CRDB_MCP_ALLOW_PASSWORD_AUTH`](#crdb-mcp-allow-password-auth)`=true`. Certificate-based authentication is recommended instead. | No | |
| `CRDB_SSL_MODE` | `require`, `verify-ca`, or `verify-full`. TLS-optional modes (`disable`, `allow`, `prefer`) require [`CRDB_MCP_ALLOW_INSECURE_DB`](#crdb-mcp-allow-insecure-db)`=true`. | No | `verify-full` |
| `CRDB_SSL_CA_PATH` | CA certificate path. | When `CRDB_SSL_MODE` is `verify-ca` or `verify-full` (the default) | |
| `CRDB_SSL_CERTFILE` | Client certificate path. | When `CRDB_SSL_MODE` is `require`, `verify-ca`, or `verify-full` (the default) | |
| `CRDB_SSL_KEYFILE` | Client key path. | When `CRDB_SSL_MODE` is `require`, `verify-ca`, or `verify-full` (the default) | |
Alternatively, use a full connection string:
| Variable | Description |
| - | - |
| `CRDB_DATABASE_URL` | A full libpq connection string (for example, `postgresql://user@host:26257/defaultdb?sslmode=verify-full&sslcert=...`). When set, the individual variables above are ignored. A `CRDB_DATABASE_URL` that omits `sslmode` is rejected at startup. |
### Query behavior
| Variable | Description | Required | Default |
| - | - | - | - |
| `CRDB_MCP_QUERY_TIMEOUT` | Per-query timeout, as a Go duration (for example, `30s`). The server cancels a query that exceeds it. | No | `30s` |
| `CRDB_MCP_MAX_ROWS_COUNT` | Cap on the maximum `LIMIT` that list-style tools issue to CockroachDB. | No | `10000` |
| `CRDB_MCP_MAX_CONNS` | Upper bound on the connection pool size (up to 100). | No | `10` |
| `CRDB_MCP_TXN_QOS` | Default transaction quality of service: `background`, `regular`, or `critical`. Refer to the precedence notes below the table. | No | `background` |
| `CRDB_MCP_ENABLE_WRITE_QUERIES` | Enable the [write tools](#write-tools). For per-operation control, grant the connecting role only the SQL privileges you want, for example `SELECT`, `INSERT`, and `UPDATE` but not `DELETE` to allow writes without deletes. | No | `false` |
| `CRDB_MCP_ALLOW_PASSWORD_AUTH` | Opt in to password-based authentication. The server refuses to start if a password is detected from any source ([`CRDB_PWD`](#crdb-pwd), a password in [`CRDB_DATABASE_URL`](#crdb-database-url), `PGPASSWORD`, or `~/.pgpass`) unless this is set to `true`. | No | `false` |
| `CRDB_MCP_ALLOW_INSECURE_DB` | Opt in to the `sslmode` values that can run without TLS: `disable`, `allow`, `prefer`. Certificate variables are not required in these modes. Development only. | No | `false` |
MCP traffic runs at `default_transaction_quality_of_service=background` by default, so it does not contend with latency-sensitive foreground workloads. The value is chosen with the following precedence:
1. [`CRDB_MCP_TXN_QOS`](#crdb-mcp-txn-qos), if set (`background`, `regular`, or `critical`).
2. Otherwise, a `default_transaction_quality_of_service=...` query parameter in [`CRDB_DATABASE_URL`](#crdb-database-url), if present.
3. Otherwise, `background`.
### Transport
| Variable | Description | Required | Default |
| - | - | - | - |
| `CRDB_MCP_TRANSPORT` | `stdio` or `http`. | No | `stdio` |
| `CRDB_MCP_HTTP_LISTEN_ADDR` | Listen address in HTTP mode, any `host:port`. Omit the host to bind all interfaces. Examples: `:8080`, `0.0.0.0:8443`, `127.0.0.1:9090`. | No | `:8080` |
| `CRDB_MCP_BEARER_TOKEN` | Bearer token that clients must send in `Authorization: Bearer {token}`. Must be at least 16 characters. | Required for `http` mode, unless [`CRDB_MCP_ALLOW_NO_BEARER`](#crdb-mcp-allow-no-bearer)`=true` | |
| `CRDB_MCP_TLS_CERT` | PEM-encoded server certificate path. | Required for `http` mode, unless [`CRDB_MCP_ALLOW_INSECURE_HTTP`](#crdb-mcp-allow-insecure-http)`=true` | |
| `CRDB_MCP_TLS_KEY` | PEM-encoded private key path. | Required for `http` mode, unless [`CRDB_MCP_ALLOW_INSECURE_HTTP`](#crdb-mcp-allow-insecure-http)`=true` | |
| `CRDB_MCP_ALLOW_INSECURE_HTTP` | Opt in to cleartext HTTP (for reverse-proxy deployments). | No | `false` |
| `CRDB_MCP_ALLOW_NO_BEARER` | Opt in to skipping bearer authentication (for upstream-auth deployments). A startup warning is logged. Mutually exclusive with [`CRDB_MCP_BEARER_TOKEN`](#crdb-mcp-bearer-token): the server refuses to start if both are set. | No | `false` |
### Logging
| Variable | Description | Required | Default |
| - | - | - | - |
| `CRDB_MCP_LOG_LEVEL` | `debug`, `info`, `warn`, or `error`. | No | `info` |
| `CRDB_MCP_LOG_PATH` | Log file path, or `-` for stderr. No rotation; use logrotate or your orchestrator. | No | |
### Tracing (OpenTelemetry)
Tracing is opt-in. When neither of the following variables is set, no exporter is installed. When enabled, tool calls and their SQL statements are exported as spans, and server logs are exported as OpenTelemetry log records. Query text and errors are redacted before export, so literals and user data never leave the server.
| Variable | Description | Required |
| - | - | - |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP gRPC endpoint; the standard `OTEL_*` environment variables are honored. | No |
| `CRDB_MCP_OTEL_FILE` | Write traces and logs as JSON lines to this file instead (takes precedence). | No |
## Tools
The following sections list the tools that the MCP server makes available to agents.
You likely do not have to invoke these functions explicitly. Most AI tools should be able to decide which functions to call based on natural language prompts provided by the user.
### Read-only tools
The following tools are registered by default. They do not modify cluster data.
#### `list_databases`
List all databases in the cluster.
| Parameter | Description | Type | Required | Default |
| - | - | - | - | - |
| `limit` | Maximum rows to return, capped at [`CRDB_MCP_MAX_ROWS_COUNT`](#crdb-mcp-max-rows-count). | integer | No | `100` |
| `offset` | Rows to skip before returning results. | integer | No | `0` |
#### `list_tables`
List all tables in a database.
| Parameter | Description | Type | Required | Default |
| - | - | - | - | - |
| `database` | Database whose tables are listed. | string | Yes | |
| `limit` | Maximum rows to return, capped at [`CRDB_MCP_MAX_ROWS_COUNT`](#crdb-mcp-max-rows-count). | integer | No | `100` |
| `offset` | Rows to skip before returning results. | integer | No | `0` |
#### `get_table_schema`
Get detailed schema information for a table, including columns and indexes. Returns the `CREATE TABLE` statement.
| Parameter | Description | Type | Required | Default |
| - | - | - | - | - |
| `database` | Database containing the table. | string | Yes | |
| `table` | Table name. | string | Yes | |
| `schema` | Schema name. | string | No | `public` |
#### `get_cluster`
Return cluster identity and version metadata: `cluster_id`, `cluster_name`, `binary_version`, `active_version`.
The identity fields (`cluster_id`, `cluster_name`, and `active_version`) require access to `crdb_internal` built-in functions, which are gated by default on CockroachDB v25.4 and later. When these fields cannot be accessed, they are returned as `null`, with the reason reported under an `unavailable` key. `binary_version` is always returned.
This tool takes no parameters.
#### `list_sql_users`
List SQL users defined in the cluster.
| Parameter | Description | Type | Required | Default |
| - | - | - | - | - |
| `limit` | Maximum rows to return, capped at [`CRDB_MCP_MAX_ROWS_COUNT`](#crdb-mcp-max-rows-count). | integer | No | `100` |
| `offset` | Rows to skip before returning results. | integer | No | `0` |
#### `list_cluster_nodes`
List cluster nodes with address, liveness, and locality. Requires `admin` or the `VIEWCLUSTERMETADATA` system privilege.
This tool takes no parameters.
#### `show_running_queries`
List in-flight SQL statements on the cluster, ordered by start time descending. Viewing other users' statements requires `admin` or the `VIEWACTIVITY` or `VIEWACTIVITYREDACTED` system privilege; otherwise, only the connecting user's own statements are returned.
| Parameter | Description | Type | Required | Default |
| - | - | - | - | - |
| `limit` | Maximum rows to return, capped at [`CRDB_MCP_MAX_ROWS_COUNT`](#crdb-mcp-max-rows-count). | integer | No | `100` |
| `offset` | Rows to skip before returning results. | integer | No | `0` |
#### `select_query`
Execute a single `SELECT` statement. Non-`SELECT` statements are rejected. A default `LIMIT` of `100` is appended when none is supplied, capped at [`CRDB_MCP_MAX_ROWS_COUNT`](#crdb-mcp-max-rows-count). An explicit `LIMIT` that exceeds [`CRDB_MCP_MAX_ROWS_COUNT`](#crdb-mcp-max-rows-count), or `LIMIT ALL`, is rejected.
| Parameter | Description | Type | Required |
| - | - | - | - |
| `query` | A single `SELECT` statement. | string | Yes |
#### `explain_query`
Return the `EXPLAIN` plan for a single SQL statement without executing it. `EXPLAIN ANALYZE` (and `EXPLAIN ANALYZE (DEBUG)`) is rejected because it executes the statement. Display options (`VERBOSE`, `DISTSQL`, `TYPES`, `OPT`, and so on) pass through.
| Parameter | Description | Type | Required |
| - | - | - | - |
| `query` | A single SQL statement. | string | Yes |
#### `show_statement`
Execute a single `SHOW` statement such as `SHOW SCHEMAS`, `SHOW INDEXES`, `SHOW REGIONS`, or `SHOW JOBS`. A default `LIMIT` is appended, capped at [`CRDB_MCP_MAX_ROWS_COUNT`](#crdb-mcp-max-rows-count).
| Parameter | Description | Type | Required | Default |
| - | - | - | - | - |
| `query` | A single `SHOW` statement. | string | Yes | |
| `limit` | Maximum rows to return, capped at [`CRDB_MCP_MAX_ROWS_COUNT`](#crdb-mcp-max-rows-count). | integer | No | `100` |
| `offset` | Rows to skip before returning results. | integer | No | `0` |
### Write tools
The following tools are registered only when [`CRDB_MCP_ENABLE_WRITE_QUERIES`](#crdb-mcp-enable-write-queries)`=true` is set. Any write tool that includes a `RETURNING` clause returns the affected rows instead of just the count.
#### `create_database`
Create a database.
| Parameter | Description | Type | Required | Default |
| - | - | - | - | - |
| `name` | Database name to create. | string | Yes | |
| `if_not_exists` | When `true`, the statement is a no-op if the database already exists. | boolean | No | `false` |
#### `create_table`
Execute a single `CREATE TABLE` statement.
| Parameter | Description | Type | Required |
| - | - | - | - |
| `statement` | A single `CREATE TABLE` statement. Qualify the table with a database when not using the connection's default database. | string | Yes |
#### `insert_rows`
Execute a single `INSERT` statement. Returns the number of rows affected.
| Parameter | Description | Type | Required |
| - | - | - | - |
| `statement` | A single `INSERT` statement. A `RETURNING` clause returns the inserted rows. Qualify the table with a database when not using the connection's default database. | string | Yes |
#### `update_rows`
Execute a single `UPDATE` statement. A `WHERE` clause is mandatory. Returns the number of rows affected.
| Parameter | Description | Type | Required |
| - | - | - | - |
| `statement` | A single `UPDATE` statement. A `WHERE` clause is mandatory. A `RETURNING` clause returns the updated rows. Qualify the table with a database when not using the connection's default database. | string | Yes |
#### `delete_rows`
Execute a single `DELETE` statement. A `WHERE` clause is mandatory. Returns the number of rows affected.
| Parameter | Description | Type | Required |
| - | - | - | - |
| `statement` | A single `DELETE` statement. A `WHERE` clause is mandatory. A `RETURNING` clause returns the deleted rows. Qualify the table with a database when not using the connection's default database. | string | Yes |
## See also
* CockroachDB MCP Server
* Install, Configure, and Connect to the CockroachDB MCP Server
* [CockroachDB MCP Server on GitHub](https://github.com/cockroachdb/cockroachdb-mcp-server)
# CockroachDB Operator Overview
Source: https://docs.cockroachlabs.com/docs/v26.3/cockroachdb-operator-overview
The CockroachDB operator is a fully-featured [Kubernetes operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) that allows you to deploy and manage CockroachDB self-hosted clusters.
## CockroachDB operator
This section describes how to:
* Deploy a CockroachDB cluster using the CockroachDB operator.
* Migrate from an existing CockroachDB Kubernetes deployment using Helm or the Public operator.
* Operate a CockroachDB cluster:
* Manage pod scheduling.
* Manage cluster resources.
* Manage certificates.
* Scale a cluster.
* Monitor a cluster.
* Upgrade a cluster.
* Override deployment templates.
* Improve cluster performance.
## Kubernetes terminology
Before starting, review some basic Kubernetes terminology. Note that CockroachDB nodes are distinct from Kubernetes "nodes" or "worker nodes".
| Feature | Description |
| - | - |
| [node](https://kubernetes.io/docs/concepts/architecture/nodes/) | A physical or virtual machine. In the deployment guide, you'll create instances and join them as worker nodes into a single Kubernetes cluster. |
| [pod](http://kubernetes.io/docs/user-guide/pods/) | A pod is a group of one of more Docker containers. In the deployment guide, each pod will run on a separate Kubernetes worker node and include one Docker container running a single CockroachDB node, reflecting our topology recommendations. |
| [operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) | An operator is an extension to Kubernetes that uses custom resources to efficiently manage specific applications. The CockroachDB operator includes two custom resource definitions to manage CockroachDB, `CrdbCluster` and `CrdbNode`. Unlike the older [](https://github.com/cockroachdb/cockroach-operator), the CockroachDB operator does not use StatefulSets and is designed to simplify multi-region deployments. |
| [persistent volume](http://kubernetes.io/docs/user-guide/persistent-volumes/) | A persistent volume is a piece of networked storage (Persistent Disk on GCE, Elastic Block Store on AWS) mounted into a pod. The lifetime of a persistent volume is decoupled from the lifetime of the pod that's using it, ensuring that each CockroachDB node binds back to the same storage on restart.
The deployment guide assumes that dynamic volume provisioning is available. When that is not the case, [persistent volume claims](http://kubernetes.io/docs/user-guide/persistent-volumes/#persistentvolumeclaims) need to be created manually. |
| [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) | RBAC, or Role-Based Access Control, is the system Kubernetes uses to manage permissions within the cluster. In order to take an action (e.g., `get` or `create`) on an API resource (e.g., a `pod`), the client must have a `Role` that allows it to do so. The deployment guide creates the RBAC resources necessary for CockroachDB to create and access certificates. |
# Resource Management with the CockroachDB Operator
Source: https://docs.cockroachlabs.com/docs/v26.3/configure-cockroachdb-operator
This page explains how to configure Kubernetes cluster resources such as memory, CPU, and storage.
On a production cluster, the resources you allocate to CockroachDB should be proportionate to your machine types and workload. Cockroach Labs recommends that you determine and set these values before deploying the cluster, but you can also update the values on a running cluster.
Run `kubectl describe nodes` to see the available resources on the instances that you have provisioned.
## Memory and CPU
You can set the CPU and memory resources allocated to the CockroachDB container on each pod.
1 CPU in Kubernetes is equivalent to 1 vCPU or 1 hyperthread. For best practices on provisioning CPU and memory for CockroachDB, refer to the Production Checklist.
Specify CPU and memory values in `cockroachdb.crdbCluster.podTemplate.spec.resources.limits` and `cockroachdb.crdbCluster.podTemplate.spec.resources.requests` in the values file used to deploy the cluster:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroachdb:
crdbCluster:
podTemplate:
spec:
resources:
limits:
cpu: 4000m
memory: 16Gi
requests:
cpu: 4000m
memory: 16gi
```
Apply the new settings to the cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
# Generate values.yaml only if it does not already exist locally, then edit it before upgrading
[ -f values.yaml ] || helm show values cockroachdb-v2/cockroachdb-chart --version {CHART_VERSION} > values.yaml
# Apply the edited values.yaml
helm upgrade $CRDBCLUSTER cockroachdb-v2/cockroachdb-chart --version {CHART_VERSION} -f values.yaml -n $NAMESPACE
```
Cockroach Labs recommends using identical values for `resources.requests` and `resources.limits`. When setting the new values, note that not all of a pod's resources will be available to the CockroachDB container. This is because a fraction of the CPU and memory is reserved for Kubernetes.
If no resource limits are specified, the pods will be able to consume the maximum available CPUs and memory. However, to avoid overallocating resources when another memory-intensive workload is on the same instance, always set resource requests and limits explicitly.
For more information on how Kubernetes handles resources, see the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/).
### Cache and SQL memory size
Each CockroachDB node reserves a portion of its available memory for its cache and for storing temporary data for SQL queries. For more information on these settings, see the Production Checklist.
The CockroachDB operator dynamically sets cache size and SQL memory size each to 25% (the recommended percentage) of the available memory, which depends on the memory request and limit you [specified](#memory-and-cpu) for your configuration. These values can be modified by adding the `cache` or `max-sql-memory` fields to `cockroachdb.crdbCluster.flags`, which is equivalent to appending `--cache` or `--max-sql-memory` as cockroach start flags.
## Persistent storage
When you start your cluster, Kubernetes dynamically provisions and mounts a persistent volume into each pod. For more information on persistent volumes, see the [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/persistent-volumes/).
The storage capacity of each volume is set in `cockroachdb.crdbCluster.dataStore.volumeClaimTemplate.spec.resources` in the values file used to deploy the cluster:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroachdb:
crdbCluster:
dataStore:
volumeClaimTemplate:
spec:
resources:
requests:
storage: "10Gi"
```
You should provision an appropriate amount of disk storage for your workload. For recommendations on this, see the Production Checklist.
### Expand disk size
If you discover that you need more capacity, you can expand the persistent volumes on a running cluster. Increasing disk size is often beneficial for CockroachDB performance.
The volume size should only adjusted on disk types that can dynamically scale up, such as Amazon EBS volumes. Adjusting the volume size on non-dynamically scaling disks is not recommended, and instead you should horizontally scale the number of disks used.
Specify a new volume size in the values file used to deploy the cluster:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroachdb:
crdbCluster:
dataStore:
volumeClaimTemplate:
spec:
resources:
requests:
storage: "100Gi"
```
Apply the new settings to the cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
# Generate values.yaml only if it does not already exist locally, then edit it before upgrading
[ -f values.yaml ] || helm show values cockroachdb-v2/cockroachdb-chart --version {CHART_VERSION} > values.yaml
# Apply the edited values.yaml
helm upgrade $CRDBCLUSTER cockroachdb-v2/cockroachdb-chart --version {CHART_VERSION} -f values.yaml -n $NAMESPACE
```
The CockroachDB operator updates all nodes and triggers a rolling restart of the pods with the new storage capacity.
To verify that the storage capacity has been updated, run `kubectl get pvc` to view the persistent volume claims (PVCs). It will take a few minutes before the PVCs are completely updated.
## Network ports
The CockroachDB operator separates network traffic into three ports:
| Protocol | Default Port | Description | Custom Resource Field |
| - | - | - | - |
| gRPC | 26258 | Used for node connections | service.ports.grpc |
| HTTP | 8080 | Used to access the DB Console | service.ports.http |
| SQL | 26257 | Used for SQL shell access | service.ports.sql |
Specify alternate port numbers in `cockroachdb.crdbCluster.service.ports` of the 's custom resource (for example, to match the default port `5432` on PostgreSQL):
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroachdb:
crdbCluster:
service:
ports:
sql: 5432
```
Apply the new settings to the cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
# Generate values.yaml only if it does not already exist locally, then edit it before upgrading
[ -f values.yaml ] || helm show values cockroachdb-v2/cockroachdb-chart --version {CHART_VERSION} > values.yaml
# Apply the edited values.yaml
helm upgrade $CRDBCLUSTER cockroachdb-v2/cockroachdb-chart --version {CHART_VERSION} -f values.yaml -n $NAMESPACE
```
The CockroachDB operator updates all nodes and triggers a rolling restart of the pods with the new port settings.
## Ingress
You can configure an [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) object to expose an internal HTTP or SQL [`ClusterIP` service](https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types) through a hostname.
In order to use the Ingress resource, your cluster must be running an [Ingress controller](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/) for load balancing. This is **not** handled by the CockroachDB operator and must be deployed separately.
Specify Ingress objects in `cockroachdb.crdbCluster.service.ingress`. Set `ingress.enabled` to `true` and specify `ingress.ui` (HTTP) or `ingress.sql` (SQL) in the values file used to deploy the cluster:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroachdb:
crdbCluster:
service:
ingress:
enabled: true
ui:
ingressClassName: nginx
annotations:
key: value
host: ui.example.com
sql:
ingressClassName: nginx
annotations:
key: value
host: sql.example.com
```
* `ingressClassName` specifies the [`IngressClass`](https://kubernetes.io/docs/concepts/services-networking/ingress/#ingress-class) of the Ingress controller. This example uses the [nginx](https://kubernetes.github.io/ingress-nginx/) controller.
* The `host` must be made publicly accessible. For example, create a route in [Amazon Route 53](https://aws.amazon.com/route53/), or add an entry to `/etc/hosts` that maps the IP address of the Ingress controller to the hostname.
Multiple hosts can be mapped to the same Ingress controller IP.
* TCP connections for SQL clients must be enabled for the Ingress controller. For an example, see the [nginx documentation](https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/).
Changing the SQL Ingress `host` on a running deployment will cause a rolling restart of the cluster, due to new node certificates being generated for the SQL host.
# Connect to a CockroachDB Cluster
Source: https://docs.cockroachlabs.com/docs/v26.3/connect-to-the-database
This page documents the required connection configuration for fully-supported third-party tools.
For a list of all supported cluster connection parameters, see the `cockroach` Connection Parameters.
For a list of community-supported third-party tools, see Third-Party Tools Supported by the Community. CockroachDB supports both native drivers and the PostgreSQL wire protocol. Most client drivers and ORM frameworks connect to CockroachDB like they connect to PostgreSQL.
## Step 1. Select your deployment
To connect to a CockroachDB Cloud cluster, you need a general connection string or connection parameters, which include the username, host, database, and port. To find these, open the **Connect** dialog for your cluster in the [CockroachDB Cloud Console](https://cockroachlabs.cloud) and select either **General connection string** or **Parameters only** as the option.
To connect to a CockroachDB Cloud cluster, you need a general connection string or connection parameters, which include the username, host, database, and port. To find these, open the **Connect** dialog for your cluster in the [CockroachDB Cloud Console](https://cockroachlabs.cloud) and select either **General connection string** or **Parameters only** as the option.
To connect to a CockroachDB self-hosted cluster, you need the general connection string or connection parameters for your cluster.
The connection strings and parameters for your cluster are output when you start the cluster.
## Step 2. Select your language and driver
### Connection string format
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full
```
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}
```
```
postgresql://{username}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `DATABASE_URL` environment variable to your connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$env:DATABASE_URL = "{connection string}"
```
### Connect to the cluster
To connect to CockroachDB with [node-postgres](https://node-postgres.com), create a new [`Client`](https://node-postgres.com/apis/client) object with a connection string.
For example:
```js theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
const { Client } = require('pg')
const client = new Client(process.env.DATABASE_URL)
client.connect()
```
Where `DATABASE_URL` is an environment variable set to a valid CockroachDB connection string.
For more information about connecting with node-postgres, see the [official node-postgres documentation](https://node-postgres.com/features/connecting).
### Connection string format
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full
```
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}
```
```
postgresql://{username}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `DATABASE_URL` environment variable to your connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$env:DATABASE_URL = "{connection string}"
```
### Connect to the cluster
To connect to CockroachDB with [Sequelize](https://sequelize.org), create a `Sequelize` object with the [CockroachDB Sequelize adapter](https://github.com/cockroachdb/sequelize-cockroachdb).
For example:
```js theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
const Sequelize = require("sequelize-cockroachdb");
const connectionString = process.env.DATABASE_URL
const sequelize = new Sequelize(connectionString)
```
Where `DATABASE_URL` is an environment variable set to a valid CockroachDB connection string.
To connect to CockroachDB with Sequelize, you must install the [CockroachDB Sequelize adapter](https://github.com/cockroachdb/sequelize-cockroachdb).
For more information about connecting with Sequelize, see the [official Sequelize documentation](https://sequelize.org/docs/v6/).
### Connection string format
```
postgresql://{username}:{password}@{host}:{port}/{database}
```
TypeORM configures SSL separately in the DataSource object (`ssl: true`).
```
postgresql://{username}:{password}@{host}:{port}/{database}
```
TypeORM configures SSL separately in the DataSource object:
```ts theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
ssl: {
ca: process.env.CA_CERT
}
```
Where `CA_CERT` is an environment variable set to the root certificate downloaded from the CockroachDB Cloud Console.
```
postgresql://{username}@{host}:{port}/{database}
```
TypeORM configures SSL separately in the DataSource object:
```ts theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
ssl: {
ca: process.env.CA_CERT,
key: process.env.CLIENT_KEY,
cert: process.env.CLIENT_CERT
}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `DATABASE_URL` environment variable to your connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$env:DATABASE_URL = "{connection string}"
```
### Connect to the cluster
To connect to CockroachDB with [TypeORM](https://typeorm.io), update your project's [`DataSource`](https://typeorm.io/data-source) with the required connection properties.
For example, suppose that you are defining the `DataSource` for your application in a file named `datasource.ts`:
```ts theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export const AppDataSource = new DataSource({
type: "cockroachdb",
url: process.env.DATABASE_URL,
ssl: true,
...
});
```
```ts theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export const AppDataSource = new DataSource({
type: "cockroachdb",
url: process.env.DATABASE_URL,
ssl: {
ca: process.env.CA_CERT
},
...
});
```
Where `CA_CERT` is an environment variable set to the root certificate downloaded from the CockroachDB Cloud Console.
```ts theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export const AppDataSource = new DataSource({
type: "cockroachdb",
url: process.env.DATABASE_URL,
ssl: {
ca: process.env.CA_CERT,
key: process.env.CLIENT_KEY,
cert: process.env.CLIENT_CERT
},
...
});
```
Where:
* `CA_CERT` is an environment variable set to the root certificate. You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert.
* `CLIENT_KEY` is an environment variable set to the client key for the user connecting to the cluster. You can generate this key with `cockroach cert create-client`.
* `CLIENT_CERT` is an environment variable set to the client certificate for the user connecting to the cluster. You can generate this certificate with `cockroach cert create-client`.
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
You can then import the `AppDataSource` into any file in your project and call `AppDataSource.initialize()` to connect to CockroachDB:
```ts theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
AppDataSource.initialize()
.then(async () => {
// Execute operations
});
```
For more information about connecting with TypeORM, see the [official TypeORM documentation](https://typeorm.io/#/).
### Connection string format
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full
```
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}
```
```
postgresql://{username}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `DATABASE_URL` environment variable to your connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$env:DATABASE_URL = "{connection string}"
```
### Connect to the cluster
To connect to CockroachDB with [Prisma](https://prisma.io/), set the `url` field of the `datasource` block in your Prisma schema to your database connection URL:
```js theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
generator client {
provider = "prisma-client-js"
}
datasource db {
provider = "cockroachdb"
url = env("DATABASE_URL")
}
model Widget {
id String @id @default(dbgenerated("gen_random_uuid()")) @db.Uuid
}
```
Where `DATABASE_URL` is an environment variable set to a valid CockroachDB connection string.
For more information about connecting with Prisma, see the [official Prisma documentation](https://www.prisma.io/cockroachdb).
To connect to a CockroachDB Basic or Standard cluster from a Python application, you must have a valid CA certificate located at `~/.postgresql/root.crt`.
For instructions on downloading a CA certificate from the CockroachDB Cloud Console, see [Connect to a CockroachDB Basic Cluster](https://www.cockroachlabs.com/docs/cockroachcloud/connect-to-a-basic-cluster) or [Connect to a CockroachDB Standard Cluster](https://www.cockroachlabs.com/docs/cockroachcloud/connect-to-your-cluster).
### Connection string format
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full
```
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}
```
```
postgresql://{username}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `DATABASE_URL` environment variable to your connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
set DATABASE_URL={connection string}
```
### Connect to the cluster
To connect to CockroachDB with [Psycopg2](https://www.psycopg.org), pass a connection string to the [`psycopg2.connect` function](https://www.psycopg.org/docs/connection.html).
For example:
```python theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
import psycopg2
import os
conn = psycopg2.connect(os.environ['DATABASE_URL'])
```
Where `DATABASE_URL` is an environment variable set to a valid CockroachDB connection string.
For more information about connecting with Psycopg, see the [official Psycopg documentation](https://www.psycopg.org/docs).
### Connection string format
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full
```
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}
```
```
postgresql://{username}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `DATABASE_URL` environment variable to your connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
set DATABASE_URL={connection string}
```
### Connect to the cluster
To connect to CockroachDB with [Psycopg3](https://www.psycopg.org), pass a connection string to the [`psycopg.connect` function](https://www.psycopg.org/psycopg3/docs/basic/usage.html).
For example:
```python theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
import psycopg
import os
with psycopg.connect(os.environ['DATABASE_URL']) as conn:
# application logic here
```
Where `DATABASE_URL` is an environment variable set to a valid CockroachDB connection string.
For more information about connecting with Psycopg, see the [official Psycopg documentation](https://www.psycopg.org/psycopg3/docs/basic/index.html).
### Connection string format
```
cockroachdb://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full
```
```
cockroachdb://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}
```
```
cockroachdb://{username}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `DATABASE_URL` environment variable to your connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
set DATABASE_URL={connection string}
```
### Connect to the cluster
To connect to CockroachDB with [SQLAlchemy](http://docs.sqlalchemy.org/), [create an `Engine` object](https://docs.sqlalchemy.org/core/engines.html) by passing the connection string to the `create_engine` function.
For example:
```python theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
from sqlalchemy import create_engine
import os
engine = create_engine(os.environ['DATABASE_URL'])
engine.connect()
```
Where `DATABASE_URL` is an environment variable set to a valid CockroachDB connection string.
To connect to CockroachDB with SQLAlchemy, you must install the [CockroachDB SQLAlchemy adapter](https://github.com/cockroachdb/sqlalchemy-cockroachdb).
For more information about connecting with SQLAlchemy, see the [official SQLAlchemy documentation](https://docs.sqlalchemy.org/core/engines_connections.html).
In order for SQLAlchemy to use the CockroachDB adapter, the connection string must begin with `cockroachdb://`. You can use the following code to modify the general connection string, which begins with `postgresql://`, to the format that works with SQLAlchemy and the CockroachDB adapter:
```python theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
engine = create_engine(os.environ['DATABASE_URL'].replace("postgresql://", "cockroachdb://"))
engine.connect()
```
### Connection string format
Django does not use a connection string. Instead, you configure connection parameters in the `DATABASES` property in your project's `settings.py` file.
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Connect to the cluster
To connect to CockroachDB from a [Django](https://www.djangoproject.com) application, update the `DATABASES` property in the project's `settings.py` file.
```python theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
## settings.py
...
DATABASES = {
'default': {
'ENGINE': 'django_cockroachdb',
'NAME': '{database}',
'USER': '{username}',
'PASSWORD': '{password}',
'HOST': '{host}',
'PORT': '{port}',
'OPTIONS': {
'sslmode': 'verify-full'
},
},
}
...
```
```python theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
## settings.py
...
DATABASES = {
'default': {
'ENGINE': 'django_cockroachdb',
'NAME': '{database}',
'USER': '{username}',
'PASSWORD': '{password}',
'HOST': '{host}',
'PORT': '{port}',
'OPTIONS': {
'sslmode': 'verify-full',
'sslrootcert': os.path.expandvars('{root-cert}'),
},
},
}
...
```
```python theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
## settings.py
...
DATABASES = {
'default': {
'ENGINE': 'django_cockroachdb',
'NAME': '{database}',
'USER': '{username}',
'HOST': '{host}',
'PORT': '{port}',
'OPTIONS': {
'sslmode': 'verify-full',
'sslrootcert': os.path.expandvars('{root-cert}'),
'sslcert': os.path.expandvars('{client-cert}'),
'sslkey': os.path.expandvars('{client-key}'),
},
},
}
...
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
To connect to CockroachDB with Django, you must install the [CockroachDB Django adapter](https://github.com/cockroachdb/django-cockroachdb).
For more information about connecting with Django, see the [official Django documentation](https://docs.djangoproject.com/en/4.0/).
### Connection string format
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full
```
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}
```
```
postgresql://{username}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `DATABASE_URL` environment variable to your connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
set DATABASE_URL={connection string}
```
### Connect to the cluster
To connect to CockroachDB with [pgx](https://github.com/jackc/pgx), use the `pgx.Connect` function.
For example:
```go theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
package main
import (
"context"
"log"
"os"
"github.com/jackc/pgx/v4"
)
func main() {
conn, err := pgx.Connect(context.Background(), os.Getenv("DATABASE_URL"))
if err != nil {
log.Fatal(err)
}
defer conn.Close(context.Background())
}
```
For more information about connecting with pgx, see the [official pgx documentation](https://pkg.go.dev/github.com/jackc/pgx).
### Connection string format
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full
```
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}
```
```
postgresql://{username}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `DATABASE_URL` environment variable to your connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
set DATABASE_URL={connection string}
```
### Connect to the cluster
To connect to CockroachDB with [pq](https://godoc.org/github.com/lib/pq), use the [`sql.Open` function](https://go.dev/doc/tutorial/database-access).
For example:
```go theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
package main
import (
"database/sql"
"log"
"os"
_ "github.com/lib/pq"
)
func main() {
db, err := sql.Open("postgres", os.Getenv("DATABASE_URL"))
if err != nil {
log.Fatal(err)
}
defer db.Close()
}
```
For more information about connecting with pq, see the [official pq documentation](https://pkg.go.dev/github.com/lib/pq).
### Connection string format
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full
```
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}
```
```
postgresql://{username}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `DATABASE_URL` environment variable to your connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
set DATABASE_URL={connection string}
```
### Connect to the cluster
To connect to CockroachDB with [GORM](http://gorm.io), use the `gorm.Open` function, with the GORM `postgres` driver.
For example:
```go theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
package main
import (
"log"
"os"
"gorm.io/driver/postgres"
"gorm.io/gorm"
)
func main() {
db, err := gorm.Open(postgres.Open(os.Getenv("DATABASE_URL")), &gorm.Config{})
if err != nil {
log.Fatal(err)
}
}
```
For more information about connecting with GORM, see the [official GORM documentation](https://gorm.io/docs).
### Connection string format
```
jdbc:postgresql://{host}:{port}/{database}?password={password}&sslmode=verify-full&user={username}
```
```
jdbc:postgresql://{host}:{port}/{database}?user={username}&password={password}&sslmode=verify-full&sslrootcert={root-cert}
```
```
jdbc:postgresql://{host}:{port}/{database}?user={username}&sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The [URL-encoded](https://wikipedia.org/wiki/Percent-encoding) path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The [URL-encoded](https://wikipedia.org/wiki/Percent-encoding) path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The [URL-encoded](https://wikipedia.org/wiki/Percent-encoding) path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The [URL-encoded](https://wikipedia.org/wiki/Percent-encoding) path to the [PKCS#8](https://tools.ietf.org/html/rfc5208)-formatted client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client --also-generate-pkcs8-key`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `JDBC_DATABASE_URL` environment variable to your JDBC connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export JDBC_DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$env:JDBC_DATABASE_URL = "{connection string}"
```
### Connect to the cluster
To connect to CockroachDB with the [JDBC](https://jdbc.postgresql.org) driver, create a `DataSource` object ([`PGSimpleDataSource` or `PGPoolingDataSource`](https://jdbc.postgresql.org/documentation/datasource/#applications-datasource)), and set the connection string with the `setUrl` class method.
For example:
```java theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
PGSimpleDataSource ds = new PGSimpleDataSource();
ds.setUrl(System.getenv("JDBC_DATABASE_URL"));
```
Where `JDBC_DATABASE_URL` is an environment variable set to a valid JDBC-compatible CockroachDB connection string.
For more information about connecting with JDBC, see the [official JDBC documentation](https://jdbc.postgresql.org/documentation/).
### Connection string format
```
jdbc:postgresql://{host}:{port}/{database}?password={password}&sslmode=verify-full&user={username}
```
```
jdbc:postgresql://{host}:{port}/{database}?user={username}&password={password}&sslmode=verify-full&sslrootcert={root-cert}
```
```
jdbc:postgresql://{host}:{port}/{database}?user={username}&sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The [URL-encoded](https://wikipedia.org/wiki/Percent-encoding) path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The [URL-encoded](https://wikipedia.org/wiki/Percent-encoding) path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The [URL-encoded](https://wikipedia.org/wiki/Percent-encoding) path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The [URL-encoded](https://wikipedia.org/wiki/Percent-encoding) path to the [PKCS#8](https://tools.ietf.org/html/rfc5208)-formatted client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client --also-generate-pkcs8-key`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `JDBC_DATABASE_URL` environment variable to your JDBC connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export JDBC_DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$env:JDBC_DATABASE_URL = "{connection string}"
```
### Connect to the cluster
To connect to CockroachDB with [Hibernate](https://hibernate.org/orm) ORM, set the object's `hibernate.connection.url` property to a valid CockroachDB connection string.
For example, if you are bootstrapping your application with a [`ServiceRegistry`](https://docs.jboss.org/hibernate/orm/current/userguide/html_single/Hibernate_User_Guide.html#bootstrap-native-registry) object, first read from the Hibernate configuration file, and then set the `hibernate.connection.url` with the `applySetting` class method:
```java theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
StandardServiceRegistry standardRegistry = new StandardServiceRegistryBuilder()
.configure( "hibernate.cfg.xml" ).applySetting("hibernate.connection.url", System.getenv("JDBC_DATABASE_URL"))
.build();
Metadata metadata = new MetadataSources( standardRegistry )
.getMetadataBuilder()
.build();
SessionFactory sessionFactory = metadata.getSessionFactoryBuilder()
.build();
```
Where `JDBC_DATABASE_URL` is an environment variable set to a valid CockroachDB connection string.
To connect to CockroachDB with Hibernate, you must specify the CockroachDB Hibernate dialect in your `hibernate.cfg.xml` configuration file.
For more information about connecting with Hibernate, see the [official Hibernate documentation](https://hibernate.org/orm/documentation).
To connect to a CockroachDB Basic or Standard cluster from a Ruby application, you must have a valid CA certificate located at `~/.postgresql/root.crt`.
For instructions on downloading a CA certificate from the CockroachDB Cloud Console, see [Connect to a CockroachDB Basic Cluster](https://www.cockroachlabs.com/docs/cockroachcloud/connect-to-a-basic-cluster) or [Connect to a CockroachDB Standard Cluster](https://www.cockroachlabs.com/docs/cockroachcloud/connect-to-your-cluster).
### Connection string format
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full
```
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}
```
```
postgresql://{username}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `DATABASE_URL` environment variable to your connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
set DATABASE_URL={connection string}
```
### Connect to the cluster
To connect to CockroachDB with the [Ruby pg](https://rubygems.org/gems/pg) driver, use the `PG.connect` function.
For example:
```ruby theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
#!/usr/bin/env ruby
require 'pg'
conn = PG.connect(ENV['DATABASE_URL'])
```
Where `DATABASE_URL` is an environment variable set to a valid CockroachDB connection string.
For more information about connecting with pg, see the [official pg documentation](https://www.rubydoc.info/gems/pg).
### Connection string format
```
cockroachdb://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full
```
```
cockroachdb://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}
```
```
cockroachdb://{username}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `DATABASE_URL` environment variable to your connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
set DATABASE_URL={connection string}
```
### Connect to the cluster
To connect to CockroachDB with [Active Record](https://github.com/rails/rails/tree/main/activerecord) from a Rails app, update the database configuration in `config/database.yml`:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
default: &default
adapter: cockroachdb
url: <%= ENV['DATABASE_URL'] %>
...
```
Where `DATABASE_URL` is an environment variable set to a valid CockroachDB connection string.
To connect to CockroachDB with Active Record, you must install the [Active Record CockroachDB adapter](https://rubygems.org/gems/activerecord-cockroachdb-adapter).
For more information about connecting with Active Record, see the [official Active Record documentation](https://guides.rubyonrails.org/active_record_querying.html).
## See also
* Install a Driver or ORM Framework
* Connection Pooling
* `cockroach` Connection Parameters
# Client Connection Parameters
Source: https://docs.cockroachlabs.com/docs/v26.3/connection-parameters
Client applications, including `cockroach` client
commands, work by establishing a network
connection to a CockroachDB cluster. The client connection parameters
determine which CockroachDB cluster they connect to, and how to
establish this network connection.
## Supported connection parameters
Most client apps, including `cockroach` client commands, determine
which CockroachDB server to connect to using a [PostgreSQL connection
URL](#connect-using-a-url). When using a URL, a client can also
specify additional SQL-level parameters. This mode provides the most
configuration flexibility.
In addition, all `cockroach` client commands also accept [discrete
connection parameters](#connect-using-discrete-parameters) that can
specify the connection parameters separately from a URL.
## When to use a URL and when to use discrete parameters
Specifying client parameters using a URL may be more convenient during
experimentation, as it facilitates copy-pasting the connection
parameters (the URL) between different tools: the output of `cockroach
start`, other `cockroach` commands, GUI database visualizer,
programming tools, etc.
Discrete parameters may be more convenient in automation, where the
components of the configuration are filled in separately from
different variables in a script or a service manager.
## Connect using a URL
A connection URL has the following format:
```
postgres://SQL user that will own the client session. | ✗ |
| `
Note that passwords with special characters must be passed as [query string parameters](#additional-connection-parameters) (e.g., `postgres://maxroach@localhost:26257/movr?password=
Find more detail about how CockroachDB handles passwords. | ✗ |
| `current database. Defaults to `defaultdb` when using `cockroach` client commands. Drivers and ORMs may have different defaults. | ✗ |
| `
For `cockroach` commands that accept a URL, you can specify the URL with the command-line flag `--url`.
If `--url` is not specified but
the environment variable `COCKROACH_URL` is defined, the environment
variable is used. Otherwise, the `cockroach` command will use
[discrete connection parameters](#connect-using-discrete-parameters)
as described below.
The ``cockroach`
commands other than `cockroach
sql`. A warning
is currently printed if it is mistakenly specified, and
future versions of CockroachDB may return an error in that case.
### Additional connection parameters
The following additional parameters can be passed after the `?` character in the URL. After the first parameter is specified, any additional parameters must be separated by an ampersand (`&`).
| Parameter | Description | Default value |
| - | - | - |
| `application_name` | An initial value for the `application_name` session variable.
Note: For Java JDBC, use `ApplicationName`. | Empty string. |
| `sslmode` | Which type of secure connection to use: `disable`, `allow`, `prefer`, `require`, `verify-ca` or `verify-full`. See [Secure Connections With URLs](#secure-connections-with-urls) for details. | `disable` |
| `sslrootcert` | Path to the CA certificate, when `sslmode` is not `disable`. | Empty string. |
| `sslcert` | Path to the client certificate, when `sslmode` is not `disable`. | Empty string. |
| `sslkey` | Path to the client private key, when `sslmode` is not `disable`. | Empty string. |
| `password` | The SQL user's password. It is not recommended to pass the password in the URL directly.
Note that passwords with special characters must be passed as [query string parameters](#additional-connection-parameters) (e.g., `postgres://maxroach@localhost:26257/movr?password=`sql.defaults.results_buffer.size` cluster setting. Can be set as a top-level query parameter or as an `options` parameter. | |
#### Supported `options` parameters
CockroachDB supports the following `options` parameters. After the first `options` parameter is specified, any additional parameters in the same connection string must be separated by a space.
| Parameter | Description |
| - | - |
| `--cluster==session variable for the SQL session. |
| `results_buffer_size` | Default size of the buffer that accumulates results for a statement or a batch of statements before they are sent to the client. Can also be set using the `sql.defaults.results_buffer.size` cluster setting. Can be set as a top-level query parameter or as an `options` parameter. |
Note that some drivers require certain characters to be properly encoded in URL connection strings. For example, spaces in [a JDBC connection string](https://jdbc.postgresql.org/documentation/use/#connection-parameters) must be specified as `%20`.
### Secure connections with URLs
The following values are supported for `sslmode`, although only the first and the last are recommended for use.
| Parameter | Description | Recommended for use |
| - | - | - |
| `sslmode=disable` | Do not use an encrypted, secure connection at all. | Use during development. |
| `sslmode=allow` | Enable a secure connection only if the server requires it.
**Not supported in all clients.** | |
| `sslmode=prefer` | Try to establish a secure connection, but accept an insecure connection if the server does not support secure connections.
**Not supported in all clients.** | |
| `sslmode=require` | Force a secure connection. An error occurs if the secure connection cannot be established. | |
| `sslmode=verify-ca` | Force a secure connection and verify that the server certificate is signed by a known CA. | |
| `sslmode=verify-full` | Force a secure connection, verify that the server certificate is signed by a known CA, and verify that the server address matches that specified in the certificate. | Use for secure deployments. |
Some client drivers and the `cockroach` commands do not support
`sslmode=allow` and `sslmode=prefer`. Check the documentation of your
SQL driver to determine whether these options are supported.
### Convert a URL for different drivers
The subcommand `cockroach convert-url` converts a connection URL, such as those printed out by `cockroach start` or included in the online documentation, to the syntax recognized by various client drivers.
The `cockroach encode-uri` command has been deprecated and merged into `cockroach convert-url`. Use `cockroach convert-url --format crdb` instead. The `encode-uri` command will be removed in a future release.
#### Basic usage
By default, `convert-url` outputs the URL in multiple formats:
```
$ ./cockroach convert-url --url "postgres://foo/bar"
```
```
# Connection URL for libpq (C/C++), psycopg (Python), lib/pq & pgx (Go), node-postgres (JS) and most pq-compatible drivers:
postgresql://root@foo:26257/bar
# Connection DSN (Data Source Name) for Postgres drivers that accept DSNs - most drivers and also ODBC:
database=bar user=root host=foo port=26257
# Connection URL for JDBC (Java and JVM-based languages):
jdbc:postgresql://foo:26257/bar?user=root
# Direct URL to CockroachDB:
postgresql://root@foo:26257/bar
```
#### Specify output format
Use the `--format` flag to output a specific format:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroach convert-url --url "postgres://foo/bar" --format jdbc
```
```
jdbc:postgresql://foo:26257/bar?user=root
```
Supported formats:
| Format | Description |
| - | - |
| `pq` | Connection URL for libpq (C/C++), psycopg (Python), lib/pq & pgx (Go), node-postgres (JS), and most pq-compatible drivers |
| `dsn` | Data Source Name for Postgres drivers that accept DSNs and ODBC |
| `jdbc` | Connection URL for JDBC (Java and JVM-based languages) |
| `crdb` | CockroachDB-specific URL format with inlined certificates for Physical Cluster Replication and Logical Data Replication |
#### Generate URLs for replication
To generate a connection URL for Physical Cluster Replication or Logical Data Replication, use the `--format crdb` flag with the `--inline` flag to inline certificates directly into the URL:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroach convert-url --url "postgresql://{user}:{password}@{node IP}:26257" --format crdb --ca-cert {path to CA certificate} --inline
```
The `--inline` flag automatically sets `--format crdb`, so you can also use:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroach convert-url --url "postgresql://{user}:{password}@{node IP}:26257" --ca-cert {path to CA certificate} --inline
```
#### Additional flags
| Flag | Description |
| - | - |
| `--url` | Connection URL to convert (required) |
| `--format` | Output format: `pq`, `dsn`, `jdbc`, or `crdb` |
| `--inline` | Inline certificates into the URL (automatically sets `--format crdb`) |
| `--database` | Database name |
| `--user` | Username |
| `--password` | Password |
| `--cluster` | Cluster name (sets `options=-ccluster=Create Security Certificates and
Rotate Certificates.
### Example URI for a Unix domain socket
The following URI is suitable to connect to a CockroachDB cluster listening for Unix domain socket connections at `/path/to/client`:
```
postgres://root@?host=/path/to/client&port=26257
```
This specifies a connection for the `root` user to an insecure cluster listening for a socket connection (e.g., a cluster started with the `--socket-dir` flag) at `/path/to/client`, and on port 26257.
### Example URI for connecting to a database with a user-defined schema
The following URI connects to a CockroachDB cluster with a user-defined schema named `max_schema` in the `movr` database using the [`options` parameter](#supported-options-parameters).
```
postgres://maxroach@db.example.com:26257/movr?sslmode=verify-full&options%3D-c%20search_path%3Dmax_schema
```
The `options=-c search_path=max_schema` parameter is URL-encoded in the example above.
## Connect using discrete parameters
Most `cockroach` commands accept connection
parameters as separate, discrete command-line flags, in addition (or
in replacement) to `--url` which [specifies all parameters as a
URL](#connect-using-a-url).
For each command-line flag that directs a connection parameter,
CockroachDB also recognizes an environment variable. The environment
variable is used when the command-line flag is not specified.
| Flag | Description |
| - | - |
| `--url` | A connection URL to use instead of the other arguments. To convert a connection URL to the syntax that works with your client driver, run `cockroach convert-url`.
**Env Variable:** `COCKROACH_URL`
**Default:** no URL |
| `--host` | The server host and port number to connect to. This can be the address of any node in the cluster.
**Env Variable:** `COCKROACH_HOST`
**Default:** `localhost:26257` |
| `--port`
`-p` | The server port to connect to. Note: The port number can also be specified via `--host`.
**Env Variable:** `COCKROACH_PORT`
**Default:** `26257` |
| `--user`
`-u` | The SQL user that will own the client session.
**Env Variable:** `COCKROACH_USER`
**Default:** `root` |
| `--insecure` | Use an insecure connection.
**Env Variable:** `COCKROACH_INSECURE`
**Default:** `false` |
| `--cert-principal-map` | A comma-separated list of `certificate directory containing the CA and client certificates and client key.
**Env Variable:** `COCKROACH_CERTS_DIR`
**Default:** `${HOME}/.cockroach-certs/` |
### Example command-line flags for an insecure connection
The following command-line flags establish an insecure connection:
```
--user=root \
--host=
When using discrete connection parameters, the file names of the CA
and client certificates and client key are derived automatically from
the value of `--certs-dir`.
## Using both URL and client parameters
Most `cockroach` commands accept both a URL and client parameters.
The information contained therein is combined in the order it appears
in the command line.
This combination is useful so that discrete command-line flags can
override settings not otherwise set in the URL.
### Example override of the current database
The `cockroach start` command prints out the following connection URL, which connects to the `defaultdb` database:
```
postgres://root@servername:26257/?sslmode=disable
```
To specify `mydb` as the current database using `cockroach sql`, run the following command:
```
cockroach sql \
--url "postgres://root@servername:26257/?sslmode=disable" \
--database mydb
```
This is equivalent to:
```
cockroach sql --url "postgres://root@servername:26257/mydb?sslmode=disable"
```
## See also
* `cockroach` Commands Overview
* Create Security Certificates
* Secure a Cluster
* Create and Manage Users
# Connection Pooling
Source: https://docs.cockroachlabs.com/docs/v26.3/connection-pooling
This page has information on planning, configuring, and using connection pools when using drivers or frameworks with CockroachDB.
In our sizing and production guidance, 1 vCPU is considered equivalent to 1 core in the underlying hardware platform.
## About connection pools
A typical database operation consists of several steps.
1. The driver uses a configuration to start a connection to the database server.
2. A network socket is opened on the client that connects to the database server.
3. Data is read or written through the network socket.
4. The connection is closed down.
5. The network socket is closed down and its resources are freed.
For simple database operations, these steps are not expensive, but as an application scales up, the performance of the application will suffer as each connection is created and destroyed. One pattern for improving performance is a connection pool, a group of already configured and established network connections between the client and database that can be reused for data operations within an application.
Each time an application reads or writes data, it will request one of the connections in the pool. After the data operation completes, the connection is returned to the pool so other operations can use it.
Connection pooling can be a enabled as a feature of the driver, a separate library used in conjunction with a driver, a feature of an application server, or a proxy server that acts as a gateway to the database server.
Sizing a connection pool is critical to maximizing application performance. Too few connections and the application will be blocked, waiting for the connection pool to establish a new connection, or for an existing connection to become available. Conversely, a connection pool with too many connections actively executing a query at the same time can also result in high latency as the cluster cycles through every connection concurrently, decreasing efficiency as the cluster incurs a cost every time it context-switches through every query. For example, the time it takes for many threads to complete many queries in parallel is typically higher than that for a smaller number of threads scaled to the number of available vCPUs or IOPs.
To read more about connection pooling, see our [What is Connection Pooling, and Why Should You Care](https://www.cockroachlabs.com/blog/what-is-connection-pooling/) blog post.
## Recommended settings for connection pools
These are Cockroach Labs recommendations for settings common to most connection pooling software. If your application has multiple workers, each worker should be configured based on the kinds of queries required for its workload.
### Set the maximum lifetime of connections
The maximum lifetime of a connection should be set to between 5 and 30 minutes. CockroachDB Cloud clusters support 30 minutes as the maximum connection lifetime. When a node is shut down or restarted, client connections can be reset after 30 minutes, causing a disruption to applications.
Cockroach Labs recommends starting with a 5 minute maximum connection lifetime and increasing the connection lifetime if there is an impact on tail latency, normally seen when there are large numbers of connections to a cluster. Setting the connection lifetime below 5 minutes is possible, but there is little benefit, and comes at a cost of increased CPU usage for clients and servers. The maximum connection lifetime changes the [rate of new connections per second](#monitor-new-connections) (i.e., average new connections per second = total connections / maximum connection age).
Configure your connection pooling with [connection jitter](#avoid-spikes-in-new-connections) to prevent connection storms.
### Set the maximum lifetime of idle connections
Set the maximum lifetime of idle connections to the same value as the [maximum lifetime of connections](#set-the-maximum-lifetime-of-connections).
### Set the maximum number of open connections
The maximum number of open connections should be set to a value that is greater than the maximum number of concurrent queries that your application worker expects to make. Doing so minimizes the need to establish new connections.
### Set the maximum number of idle connections
Set the maximum number of idle connections to the same value as the [maximum number of open connections](#set-the-maximum-number-of-open-connections).
## Optimize your connection pool
Follow these additional recommendations optimize the performance of your connection pool software.
### Avoid spikes in new connections
If possible configure your connection pool to avoid periodic spikes in new connections, also known as "thundering herds" or "connection storms." Most connection pools offer the ability to stagger their connection age through the use of a connection lifetime "jitter," where the wait time between new connections is randomized. Using a connection jitter of 10% of the maximum connection lifetime smooths out the rate of new connections to a cluster. For example, for a maximum connection lifetime of 1800 seconds (30 minutes), set the connection jitter to 500 seconds (3 minutes).
### Validate connections in a pool
After a connection pool initializes connections to CockroachDB clusters, those connections can occasionally break. This could be due to changes in the cluster topology, or rolling upgrades and restarts, or network disruptions.
Validating connections is typically handled automatically by the connection pool. For example, in HikariCP the connection is validated whenever you request a connection from the pool, and the [`keepaliveTime` property](https://github.com/brettwooldridge/HikariCP#frequently-used) allows you to configure an interval to periodically check if the connections in the pool are valid. Whatever connection pool you use, make sure connection validation is enabled when running your application.
## Size connection pools
If your connection pool is properly configured, the total number of open connections to your cluster will typically be at least 100 times larger than the number of new connections per second to your cluster.
Idle connections in CockroachDB do not consume many resources compared to PostgreSQL. Unlike PostgreSQL, which has a hard limit of 5000 connections, CockroachDB can safely support tens of thousands of connections.
The **SQL Connection** graph shows how many new connections are being created each second. The **Open SQL Sessions** graph shows the total number of SQL client connections across the cluster. To determine if your connection pool is correctly configured use the metrics from these graphs in the following formula:
**SQL Connection Attempts \< SQL Open Sessions/100**
That is, divide the number of open SQL sessions by 100. This result should be less than the number of new SQL connections per second.
Configure the minimum number of connections to equal to the maximum number of connections, creating a fixed pool size.
Configure the maximum connection lifetime to 30 minutes.
Multi-region Basic and Standard clusters have the same recommendations as single-region clusters.
Cockroach Labs strongly recommends that your application instance run in the same region as your CockroachDB cluster.
If your connection pool is properly configured, the total number of open connections to your cluster will typically be at least 100 times larger than the number of new connections per second to your cluster.
Idle connections in CockroachDB do not consume many resources compared to PostgreSQL. Unlike PostgreSQL, which has a hard limit of 5000 connections, CockroachDB can safely support tens of thousands of connections.
Use the following formula as a starting point to size the connection pool:
**active connections = (number of cores \* 4)**
If you have a large number of services connecting to the same cluster, make sure the number of [concurrent active connections](#monitor-active-connections) across all the services does not exceed this recommendation. If each service has its own connection pool, ensure the total number of concurrent active connections across all services stays within this limit.
For multi-region clusters, create a connection pool per region, and apply the same active connection limits for each region in your cluster. For example, if you have 3 regions in your cluster, and each region has 12 vCPUs, the concurrent active connection limit **per region** is 48 (12 \[processor cores] \* 4).
[Connection pool sizes](#set-the-maximum-number-of-open-connections) can be much larger than the active connections limit, since most connections will be idle and have almost no overhead.
To control the maximum number of non-superuser (`root` user or other `admin` role) connections a gateway node can have open at one time, use the `server.max_connections_per_gateway` cluster setting. If a new non-superuser connection would exceed this limit, the error message `"sorry, too many clients already"` is returned, along with error code `53300`.
Cockroach Labs strongly recommends that your application instance run in the same region as your CockroachDB cluster.
If your connection pool is properly configured, the total number of open connections to your cluster will typically be at least 100 times larger than the number of new connections per second to your cluster.
Idle connections in CockroachDB do not consume many resources compared to PostgreSQL. Unlike PostgreSQL, which has a hard limit of 5000 connections, CockroachDB can safely support tens of thousands of connections.
Use the following formula as a starting point to size the connection pool:
**active connections = (number of cores \* 4)**
If you have a large number of services connecting to the same cluster, make sure the number of [concurrent active connections](#monitor-active-connections) across all the services does not exceed this recommendation. If each service has its own connection pool, ensure the total number of concurrent active connections across all services stays within this limit.
For multi-region clusters, create a connection pool per region, and apply the same active connection limits for each region in your cluster. For example, if you have 3 regions in your cluster, and each region has 12 vCPUs, the concurrent active connection limit **per region** is 48 (12 \[processor cores] \* 4).
[Connection pool sizes](#set-the-maximum-number-of-open-connections) can be much larger than the active connections limit, since most connections will be idle and have almost no overhead.
To control the maximum number of non-superuser (`root` user or other `admin` role) connections a gateway node can have open at one time, use the `server.max_connections_per_gateway` cluster setting. If a new non-superuser connection would exceed this limit, the error message `"sorry, too many clients already"` is returned, along with error code `53300`.
Cockroach Labs strongly recommends that your application instance run in the same region as your CockroachDB cluster.
## Monitor SQL connections
To validate that your connection pool is correctly configured, monitor the SQL connection metrics and graphs on your cluster.
### Monitor new connections
The **SQL Connections** graph shows the number of new SQL connections per second.
A misconfigured connection pool will result in most database operations requiring a new connection to be established, which will increase query latency.
### Monitor active connections
The **Open SQL Sessions** graph shows the number of open connections on your cluster.
If your connection pool is properly configured, the total number of open connections to your cluster should be at least 100 times larger than the number of new connections per second to your cluster.
To validate that your connection pool is correctly configured, monitor the SQL connection metrics and graphs on your cluster.
### Monitor new connections
The `sql.new_conns` metric and **SQL Connection Rate** graph expose the number of new SQL connections per second.
A misconfigured connection pool will result in most database operations requiring a new connection to be established, which will increase query latency.
### Monitor active connections
The `sql.conns` metric and **Open SQL Sessions** graph show the number of open connections on your cluster or node.
The `sql.statements.active` metric and **Active SQL Statements** graph show the number of **active** connections on your cluster or node. A connection is "active" when it is actively executing a query.
Using the following formula:
**active connections = (number of cores \* 4)**
If the number of concurrent active connections exceeds 4 times the number of cores in your cluster, your application is likely not achieving maximum throughput. You should reduce the concurrent workload across your applications, or scale the cluster by adding more nodes or cores per node. Reducing the number of active connections may increase overall throughput, possibly at the expense of increased tail latency for your queries.
If your connection pool is properly configured, the total number of open connections to your cluster should be at least 100 times larger than the number of new connections per second to your cluster.
To validate that your connection pool is correctly configured, monitor the SQL connection metrics and graphs on your cluster.
### Monitor new connections
The `sql.new_conns` metric and **SQL Connection Rate** graph expose the number of new SQL connections per second.
A misconfigured connection pool will result in most database operations requiring a new connection to be established, which will increase query latency.
### Monitor active connections
The `sql.conns` metric and **Open SQL Sessions** graph show the number of open connections on your cluster or node.
The `sql.statements.active` metric and **Active SQL Statements** graph show the number of **active** connections on your cluster or node. A connection is "active" when it is actively executing a query.
Using the following formula:
**active connections = (number of cores \* 4)**
If the number of concurrent active connections exceeds 4 times the number of cores in your cluster, your application is likely not achieving maximum throughput. You should reduce the concurrent workload across your applications, or scale the cluster by adding more nodes or cores per node. Reducing the number of active connections may increase overall throughput, possibly at the expense of increased tail latency for your queries.
If your connection pool is properly configured, the total number of open connections to your cluster should be at least 100 times larger than the number of new connections per second to your cluster.
## Serverless functions
"Serverless" refers to stateless, programmatic functions deployed in a cloud environment that provides an execution framework to provision resources dynamically, such as Amazon Lambda functions.
If your application uses serverless functions to connect to CockroachDB, use a connection pool if you plan to invoke functions frequently. To ensure that the connection pool is reused across invocations of the same function instance, initialize the connection pool variable outside the scope of the serverless function definition. Set the maximum connection pool size to 1, unless your function is multi-threaded and establishes multiple concurrent requests to your database within a single function instance.
## Example
In this example, a Java application similar to the basic JDBC example uses the PostgreSQL JDBC driver and HikariCP as the connection pool layer to connect to a CockroachDB cluster. The database is being run on 10 cores across the cluster.
Using the connection pool formula above:
**connections = (10 \[processor cores] \* 4)**
The connection pool size should be 40.
The maximum lifetime of a connection is set to 300000 milliseconds, or 5 minutes.
```java theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
HikariConfig config = new HikariConfig();
config.setJdbcUrl("jdbc:postgresql://localhost:26257/bank");
config.setUsername("maxroach");
config.setPassword("password");
config.addDataSourceProperty("ssl", "true");
config.addDataSourceProperty("sslmode", "require");
config.addDataSourceProperty("reWriteBatchedInserts", "true");
config.setAutoCommit(false);
config.setMaximumPoolSize(40);
config.setMaxLifetime(300000);
config.setKeepaliveTime(150000);
HikariDataSource ds = new HikariDataSource(config);
Connection conn = ds.getConnection();
```
In this example, a Go application similar to the basic pgx example uses the pgxpool library to create a connection pool on a CockroachDB cluster. The database is being run on 10 cores across the cluster.
Using the connection pool formula above:
**connections = (10 \[processor cores] \* 4)**
The connection pool size should be 40, and is set using the `pool_max_conns` query parameter.
The `pool_max_conn_lifetime` query parameter sets the maximum age of a connection to 300 seconds, or 5 minutes.
The `pool_max_conn_lifetime_jitter` query parameter sets the connection jitter to 30 seconds, or 10% of the maximum connection age.
```go theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
// Set connection pool configuration, with maximum connection pool size.
config, err := pgxpool.ParseConfig("postgres://max:roach@127.0.0.1:26257/bank?sslmode=require&pool_max_conns=40&pool_max_conn_lifetime=300s&pool_max_conn_lifetime_jitter=30s")
if err != nil {
log.Fatal("error configuring the database: ", err)
}
// Create a connection pool to the "bank" database.
dbpool, err := pgxpool.ConnectConfig(context.Background(), config)
if err != nil {
log.Fatal("error connecting to the database: ", err)
}
defer dbpool.Close()
```
For a full list of connection pool configuration parameters for pgxpool, see the pgxpool documentation.
## Known limitations
* When a node has both a high number of client connections and running queries, the node may crash due to memory exhaustion. This is due to CockroachDB not accurately limiting the number of clients and queries based on the amount of available RAM on the node. To prevent memory exhaustion, monitor each node's memory usage and ensure there is some margin between maximum CockroachDB memory usage and available system RAM. For more details about memory usage in CockroachDB, see [this blog post](https://www.cockroachlabs.com/blog/memory-usage-cockroachdb/).
# Data Resilience
Source: https://docs.cockroachlabs.com/docs/v26.3/data-resilience
CockroachDB provides built-in [**high availability (HA)**](#high-availability) features and [**disaster recovery (DR)**](#disaster-recovery) tooling to achieve operational resiliency in various deployment topologies and use cases.
* HA features ensure continuous access to data without interruption even in the presence of failures or disruptions to maximize uptime.
* DR tools allow for recovery from major incidents to minimize downtime and data loss.
![Diagram showing how HA features and DR tools create a resilient CockroachDB deployment.]()
You can balance required SLAs and recovery objectives with the cost and management of each of these features to build a resilient deployment.
* **Recovery Point Objective (RPO)**: The maximum amount of data loss (measured by time) that an organization can tolerate.
* **Recovery Time Objective (RTO)**: The maximum length of time it should take to restore normal operations following an outage.
For a practical guide on how CockroachDB uses Raft to replicate, distribute, and rebalance data, refer to the CockroachDB Resilience demo.
## High availability
* **Multi-active availability**: CockroachDB's built-in Raft replication stores data safely and consistently on multiple nodes to ensure no downtime even during a temporary node outage. Replication controls allow you to configure the number and location of replicas to suit a deployment.
* For more detail on planning for single-region or multi-region recovery, refer to Single-region survivability planning or Multi-region survivability planning.
* **Advanced fault tolerance**: Capabilities built in to CockroachDB to perform routine maintenance operations with minimal impact to foreground performance. For example, online schema changes, write-ahead log failover, and Leader leases.
* **Logical data replication (LDR)** (Preview): A cross-cluster replication tool between active CockroachDB clusters, which supports a range of topologies. LDR provides eventually consistent, table-level replication between the clusters. Individually, each active cluster uses CockroachDB multi-active availability to achieve low, single-region write latency with transactionally consistent writes using Raft replication.
### Choose an HA strategy
| | Single-region replication (synchronous) | Multi-region replication (synchronous) | Logical data replication (asynchronous) |
| - | - | - | - |
| **RPO** | 0 seconds | 0 seconds | Immediate mode 0.5 seconds |
| **RTO** | Zero RTO Potential increased latency for 1-9 seconds | Zero RTO Potential increased latency for 1-9 seconds | Zero RTO Application traffic failover time |
| **Write latency** | Region-local write latency p50 latency \< 5ms (for multiple availability zones in us-east1) | Cross-region write latency p50 latency > 50ms (for a multi-region cluster in us-east1, us-east-2, us-west-1) | Region-local latency depending on design p50 latency \< 5ms (for multiple availability zones in us-east1) |
| **Recovery** | Automatic | Automatic | Semi-automatic |
| **Fault tolerance** | Zero RPO node, availability zone failures within a cluster | Zero RPO node, availability zone failures, region failures within a cluster | Zero RPO node, availability zone within a cluster, region failures with loss up to RPO in a two-region (or two datacenter) setup |
| **Minimum regions to achieve fault tolerance** | 1 | 3 | 2 |
For details on designing your cluster topology for HA with replication, refer to the Disaster Recovery Planning page.
## Disaster recovery
* **Backup and point-in-time restore**: Point-in-time backup and restore allows you to roll back to a specific point in time. Multiple supported cloud storage providers means that you can store backups in your chosen provider. Incremental backups allow you to configure backup frequency for lower RPO.
* **Physical cluster replication (PCR)**: A cross-cluster replication tool between an active primary CockroachDB cluster and a passive standby CockroachDB cluster. PCR provides transactionally consistent full-cluster replication.
### Choose a DR strategy
CockroachDB is designed to recover automatically; however, building backups or PCR into your DR plan protects against unforeseen incidents.
| | Point-in-time backup & restore | Physical cluster replication (asynchronous) |
| - | - | - |
| **RPO** | >=5 minutes | 10s of seconds |
| **RTO** | Minutes to hours, depending on data size and number of nodes | Seconds to minutes, depending on cluster size, and time of failover |
| **Write latency** | No impact | No impact |
| **Recovery** | Manual restore | Manual failover |
| **Fault tolerance** | Not applicable | Zero RPO node, availability zone within a cluster, region failures with loss up to RPO in a two-region (or two-datacenter) setup |
| **Minimum regions to achieve fault tolerance** | 1 | 2 |
## See also
* Architecture Overview
* Set Up Logical Data Replication
* Set Up Physical Cluster Replication
* Physical Cluster Replication Technical Overview
* Backup Architecture
* Backup and Restore Overview
* Managed Backups
# CockroachDB Resilience Demo
Source: https://docs.cockroachlabs.com/docs/v26.3/demo-cockroachdb-resilience
This page describes how to perform a hands-on demonstration of CockroachDB's fault-tolerant design allowing services to remain available during a failure and recovery.
## Run a guided demo in CockroachDB Cloud
CockroachDB Cloud Advanced includes a built-in fault tolerance demo in the Cloud Console that automatically runs a sample workload and executes a controlled availability zone failure on your cluster, showing real-time metrics of query latency and failure rate during the outage and recovery.
The CockroachDB Cloud fault tolerance demo is in Preview.
The following prerequisites are needed to run the fault tolerance demo:
* You must have CockroachDB Advanced cluster with at least three nodes.
* All nodes must be healthy.
* The cluster's CPU utilization must be below 30%.
* The cluster must not currently be in a locked state as a result of maintenance such as scaling. Clusters that were recently created may be locked intermittently while post-provisioning setup completes. Monitor your cluster's status in the Cloud Console to make sure it is not in a maintenance state.
* You must have the Cluster Operator and/or Cluster Admin role assigned to start the demo.
* Another fault tolerance demo must not currently be running.
In total, the fault tolerance demo takes approximately 10 to 15 minutes to complete. The demo creates a temporary database and demo workload on your cluster, then cleans up after the demo is complete. If the demo becomes stuck, the system will automatically force a full cleanup after 30 minutes. You do not need to contact support before this window has elapsed.
The fault tolerance demo is not recommended to be run on a production cluster. The demo temporarily blocks network communication to all nodes in one availability zone. This disruption is real, and may affect cluster performance during the outage window. Instead, use a dedicated test or staging cluster for the fault tolerance demo.
To run the fault tolerance demo, open the Cloud Console and navigate to the **Overview** page, then select **Actions > Fault tolerance demo**.
### Known limitations
The following known limitations apply to the fault tolerance demo in CockroachDB Cloud:
* QPS may briefly appear to drop to zero on the chart during or after the disruption phase. This occurs because metrics from the nodes in the disrupted availability zone are temporarily unavailable during the network block, it does not reflect actual cluster behavior. The unaffected nodes continue serving traffic normally.
* If you cancel the fault tolerance demo and immediately try to start a new one, you may encounter an error. Wait a few minutes before starting again.
* Clean-up of the temporary workload and database may continue for a few minutes after the demo ends. Refresh the Console page if the message does not clear after a few minutes.
## Run a manual demo on a local machine
This guide walks you through a simple demonstration of CockroachDB's resilience on a local cluster deployment.
Starting with a 6-node local cluster with the default 3-way replication, you will perform the following steps:
1. Run a sample workload.
2. Terminate a node to simulate failure.
3. Observe the cluster continuing to serve query traffic uninterrupted during the failure state.
4. Observe the cluster repair itself by re-replicating missing data to other nodes.
5. Prepare the cluster for 2 simultaneous node failures by increasing to 5-way replication. ^.6. Take two nodes offline at the same time, and again observe how the cluster continues uninterrupted.
### Before you begin
Make sure you have already installed CockroachDB.
### Step 1. Start a 6-node cluster
1. In separate terminal windows, use the `cockroach start` command to start six nodes:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--insecure \
--store=fault-node1 \
--listen-addr=localhost:26257 \
--http-addr=localhost:8080 \
--join=localhost:26257,localhost:26258,localhost:26259
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--insecure \
--store=fault-node2 \
--listen-addr=localhost:26258 \
--http-addr=localhost:8081 \
--join=localhost:26257,localhost:26258,localhost:26259
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--insecure \
--store=fault-node3 \
--listen-addr=localhost:26259 \
--http-addr=localhost:8082 \
--join=localhost:26257,localhost:26258,localhost:26259
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--insecure \
--store=fault-node4 \
--listen-addr=localhost:26260 \
--http-addr=localhost:8083 \
--join=localhost:26257,localhost:26258,localhost:26259
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--insecure \
--store=fault-node5 \
--listen-addr=localhost:26261 \
--http-addr=localhost:8084 \
--join=localhost:26257,localhost:26258,localhost:26259
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--insecure \
--store=fault-node6 \
--listen-addr=localhost:26262 \
--http-addr=localhost:8085 \
--join=localhost:26257,localhost:26258,localhost:26259
```
2. Use the `cockroach init` command to perform a one-time initialization of the cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach init --insecure --host=localhost:26257
```
### Step 2. Set up load balancing
In this tutorial, you run a sample workload to simulate multiple client connections. Each node is an equally suitable SQL gateway for the load, but it's always recommended to spread requests evenly across nodes. This section shows how to set up the open-source [HAProxy](http://www.haproxy.org/) load balancer.
1. Install HAProxy.
If you're on a Mac and use Homebrew, run:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ brew install haproxy
```
If you're using Linux and use apt-get, run:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo apt-get install haproxy
```
2. Run the `cockroach gen haproxy` command, specifying the port of any node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach gen haproxy --insecure --host=localhost --port=26257
```
This command generates an `haproxy.cfg` file automatically configured to work with the nodes of your running cluster.
3. In `haproxy.cfg`, change `bind :26257` to `bind :26000`. This changes the port on which HAProxy accepts requests to a port that is not already in use by a node.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
sed -i.saved 's/^ bind :26257/ bind :26000/' haproxy.cfg
```
4. Start HAProxy, with the `-f` flag pointing to the `haproxy.cfg` file:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ haproxy -f haproxy.cfg &
```
### Step 3. Run a sample workload
Use the `cockroach workload` command to run CockroachDB's built-in version of the YCSB benchmark, simulating multiple client connections, each performing mixed read/write operations.
1. Load the initial `ycsb` schema and data, pointing it at HAProxy's port:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload init ycsb --splits=50 'postgresql://root@localhost:26000?sslmode=disable'
```
The `--splits` flag tells the workload to manually split ranges a number of times. This is not something you'd normally do, but for the purpose of this tutorial, it makes it easier to visualize the movement of data in the cluster.
2. Run the `ycsb` workload, pointing it at HAProxy's port:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload run ycsb \
--duration=20m \
--concurrency=3 \
--max-rate=1000 \
--tolerate-errors \
'postgresql://root@localhost:26000?sslmode=disable'
```
This command initiates 3 concurrent client workloads for 20 minutes, but limits the total load to 1000 operations per second (since you're running everything on a single machine).
You'll see per-operation statistics print to standard output every second:
```
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
1.0s 0 902.8 930.9 1.1 2.1 4.1 62.9 read
1.0s 0 46.5 48.0 3.4 4.7 6.0 6.0 update
2.0s 0 923.3 926.9 1.0 2.9 5.5 8.9 read
2.0s 0 38.0 43.0 3.0 6.0 7.6 7.6 update
3.0s 0 901.1 918.3 1.1 2.5 5.0 6.6 read
3.0s 0 55.0 47.0 3.4 7.9 9.4 11.5 update
4.0s 0 948.9 926.0 1.0 1.6 2.6 5.0 read
4.0s 0 46.0 46.7 3.1 5.2 16.8 16.8 update
5.0s 0 932.0 927.2 1.1 1.8 2.9 13.6 read
5.0s 0 56.0 48.6 3.0 4.2 5.2 5.5 update
...
```
After the specified duration (20 minutes in this case), the workload will stop and you'll see totals printed to standard output.
### Step 4. Check the workload
Initially, the workload creates a new database called `ycsb`, creates the table `public.usertable` in that database, and inserts rows into the table. Soon, the load generator starts executing approximately 95% reads and 5% writes.
1. Go to the DB Console at [http://localhost:8080](http://localhost:8080).
2. To check the SQL queries getting executed, click **Metrics** on the left, and hover over the **SQL Statements** graph at the top:
![DB Console SQL Statements]()
3. To check the client connections from the load generator, select the **SQL** dashboard and hover over the **Open SQL Sessions** graph:
![DB Console Open Sessions]()
You'll notice 3 client connections from the load generator. If you want to check that HAProxy balanced each connection to a different node, you can change the **Graph** dropdown from **Cluster** to specific nodes.
4. To see more details about the `ycsb` database and the `public.usertable` table, click **Databases** in the upper left and click **ycsb**:
![DB Console Databases]()
You can also view the schema and other table details of `public.usertable` by clicking the table name:
![DB Console usertable]()
5. By default, CockroachDB replicates all data 3 times and balances it across all nodes. To see this balance, click **Overview** and check the replica count across all nodes:
![DB Console Overview]()
### Step 5. Simulate a single node failure
When a node fails, the cluster waits for the node to remain offline for 5 minutes by default before considering it dead, at which point the cluster automatically repairs itself by re-replicating any of the replicas on the down nodes to other available nodes.
1. In a new terminal, edit the default replication zone to reduce the amount of time the cluster waits before considering a node dead to the minimum allowed of 1 minute and 15 seconds:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --insecure --host=localhost:26000 \
--execute="SET CLUSTER SETTING server.time_until_store_dead = '1m15s';"
```
2. Get the process IDs of the nodes:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
ps -ef | grep cockroach | grep -v grep
```
```
503 53645 1 0 10:18AM ttys013 0:44.80 cockroach start --insecure --store=fault-node1 --listen-addr=localhost:26257 --http-addr=localhost:8080 --join=localhost:26257,localhost:26258,localhost:26259
503 53651 1 0 10:18AM ttys013 0:38.81 cockroach start --insecure --store=fault-node2 --listen-addr=localhost:26258 --http-addr=localhost:8081 --join=localhost:26257,localhost:26258,localhost:26259
503 53671 1 0 10:18AM ttys013 0:35.40 cockroach start --insecure --store=fault-node3 --listen-addr=localhost:26259 --http-addr=localhost:8082 --join=localhost:26257,localhost:26258,localhost:26259
503 53684 1 0 10:18AM ttys013 0:38.44 cockroach start --insecure --store=fault-node4 --listen-addr=localhost:26260 --http-addr=localhost:8083 --join=localhost:26257,localhost:26258,localhost:26259
503 53708 1 0 10:18AM ttys013 0:39.49 cockroach start --insecure --store=fault-node5 --listen-addr=localhost:26261 --http-addr=localhost:8084 --join=localhost:26257,localhost:26258,localhost:26259
503 53730 1 0 10:18AM ttys013 0:33.37 cockroach start --insecure --store=fault-node6 --listen-addr=localhost:26262 --http-addr=localhost:8085 --join=localhost:26257,localhost:26258,localhost:26259
```
3. Gracefully shut down the node stored in `fault-node5`, specifying its process ID (in this example, `53708`):
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kill -TERM 53708
```
### Step 6. Check load continuity and cluster health
Go back to the DB Console, click **Metrics** on the left, and verify that the cluster as a whole continues serving data, despite one of the nodes being unavailable and marked as **Suspect**:
![DB Console Suspect Node]()
This shows that when all ranges are replicated 3 times (the default), the cluster can tolerate a single node failure because the surviving nodes have a majority of each range's replicas (2/3).
### Step 7. Watch the cluster repair itself
Click **Overview** on the left:
![DB Console Cluster Repair]()
Because you reduced the time it takes for the cluster to consider the down node dead, after 1 minute or so, the cluster will consider the down node "dead", and you'll see the replica count on the remaining nodes increase and the number of under-replicated ranges decrease to 0. This shows the cluster repairing itself by re-replicating missing replicas.
### Step 8. Prepare for two simultaneous node failures
At this point, the cluster has recovered and is ready to handle another failure. However, the cluster cannot handle two *near-simultaneous* failures in this configuration. Failures are "near-simultaneous" if they are closer together than the `server.time_until_store_dead` cluster setting plus the time taken for the number of replicas on the dead node to drop to zero. If two failures occurred in this configuration, some ranges would become unavailable until one of the nodes recovers.
To be able to tolerate 2 of 5 nodes failing simultaneously without any service interruption, ranges must be replicated 5 times.
1. In the terminal window where you started `fault-node5` initially, start it again using the same command you used to [start the node initially](#step-1-start-a-6-node-cluster):
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--insecure \
--store=fault-node5 \
--listen-addr=localhost:26261 \
--http-addr=localhost:8084 \
--join=localhost:26257,localhost:26258,localhost:26259
```
2. Use the `ALTER RANGE ... CONFIGURE ZONE` command to change the cluster's `default` replication factor to 5:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --insecure --host=localhost:26000 \
--execute="ALTER RANGE default CONFIGURE ZONE USING num_replicas=5;"
```
3. In the DB Console **Overview** dashboard, watch the replica count increase and even out across all 6 nodes:
![DB Console Cluster Restore]()
This shows the cluster up-replicating so that each range has 5 replicas, one on each node.
### Step 9. Simulate two simultaneous node failures
Gracefully shut down **2 nodes**, specifying the [process IDs you retrieved earlier](#step-5-simulate-a-single-node-failure):
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kill -TERM {process IDs}
```
### Step 10. Check cluster status and service continuity
1. Click **Overview** on the left, and verify the state of the cluster:
![DB Console Cluster Health]()
2. To verify that the cluster still serves data, use the `cockroach sql` command to interact with the cluster.
Count the number of rows in the `ycsb.public.usertable` table to see that it serves reads:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --insecure --host=localhost:26257 \
--execute="SELECT count(*) FROM ycsb.public.usertable;"
```
```
count
+-------+
10000
(1 row)
```
Insert data to see that it serves writes:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --insecure --host=localhost:26257 \
--execute="INSERT INTO ycsb.public.usertable VALUES ('asdf', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);"
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --insecure --host=localhost:26257 \
--execute="SELECT count(*) FROM ycsb.public.usertable;"
```
```
count
+-------+
10001
(1 row)
```
This shows that when all ranges are replicated 5 times, the cluster can tolerate 2 simultaneous node outages because the surviving nodes have a majority of each range's replicas (3/5).
### Step 11. Clean up
1. In the terminal where the YCSB workload is running, press **CTRL + c**.
2. Terminate HAProxy:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ pkill haproxy
```
3. Gracefully shut down the remaining **4 nodes**, specifying the [process IDs you retrieved earlier](#step-5-simulate-a-single-node-failure) and the new process ID for the node stored in `fault-node5`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kill -TERM {process IDs}
```
For the final 2 nodes, the shutdown process will take longer (about a minute each) and will eventually force the nodes to stop. This is because, with only 2 of 5 nodes left, a majority of replicas are not available, and so the cluster is no longer operational.
1. To restart the cluster at a later time, run the same `cockroach start` commands as in [Step 1. Start a 6-node cluster](#step-1-start-a-6-node-cluster) from the directory containing the nodes' data stores.
2. If you do not plan to restart the cluster, you may want to remove the nodes' data stores and the HAProxy config files:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ rm -rf fault-node1 fault-node2 fault-node3 fault-node4 fault-node5 fault-node6 haproxy.cfg haproxy.cfg.saved
```
## What's next?
Explore other CockroachDB benefits and features:
* Replication & Rebalancing
* CockroachDB Resilience
* Serializable Transactions
# Replication and Rebalancing
Source: https://docs.cockroachlabs.com/docs/v26.3/demo-replication-and-rebalancing
This page guides you through a simple demonstration of how CockroachDB replicates, distributes, and rebalances data. Starting with a 3-node local cluster, you'll write some data and verify that it replicates in triplicate by default. You'll then add 2 more nodes and watch how CockroachDB automatically rebalances replicas to efficiently use all available capacity.
## Before you begin
Make sure you have already installed CockroachDB.
## Step 1. Generate certificates
1. Create two directories:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir certs my-safe-directory
```
2. Create the CA (Certificate Authority) certificate and key pair:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-ca \
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
3. Create the certificate and key pair for your nodes:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-node \
localhost \
$(hostname) \
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
Because you're running a local cluster and all nodes use the same hostname (`localhost`), you only need a single node certificate. Note that this is different than running a production cluster, where you would need to generate a certificate and key for each node, issued to all common names and IP addresses you might use to refer to the node as well as to any load balancer instances.
4. Create a client certificate and key pair for the `root` user:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-client \
root \
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
## Step 2. Start a 3-node cluster
1. Use the `cockroach start` command to start node 1:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--store=rep-node1 \
--listen-addr=localhost:26257 \
--http-addr=localhost:8080 \
--join=localhost:26257,localhost:26258,localhost:26259
```
2. In a new terminal, start node 2:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--store=rep-node2 \
--listen-addr=localhost:26258 \
--http-addr=localhost:8081 \
--join=localhost:26257,localhost:26258,localhost:26259
```
3. In a new terminal, start node 3:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--store=rep-node3 \
--listen-addr=localhost:26259 \
--http-addr=localhost:8082 \
--join=localhost:26257,localhost:26258,localhost:26259
```
4. In a new terminal, use the `cockroach init` command to perform a one-time initialization of the cluster, sending the request to any node on the `--join` list:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach init --certs-dir=certs --host=localhost:26257
```
You'll see the following message:
```
Cluster successfully initialized
```
At this point, each node also prints helpful startup details to its log.
## Step 3. Create a SQL user
You'll use a non-`root` user for running a client workload and accessing the DB Console.
1. In the same terminal, as the `root` user, open the built-in SQL shell against any node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql \
--certs-dir=certs \
--host=localhost:26257
```
2. Create the `maxroach` user with a password:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> CREATE USER maxroach WITH PASSWORD ' GRANT admin TO maxroach;
```
This role assignment is for convenience for this tutorial; it gives the user access to all data without the need for additional privileges. For more details, see Authorization.
4. Exit the SQL shell:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> \q
```
## Step 4. Write data
1. In the same terminal, run the `cockroach workload` command to generate an example `intro` database. In the connection string, replace ` SHOW DATABASES;
```
```
database_name
+---------------+
defaultdb
intro
postgres
system
(4 rows)
```
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SHOW TABLES FROM intro;
```
```
table_name
--------------
mytable
(1 row)
```
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SELECT * FROM intro.mytable WHERE (l % 2) = 0;
```
```
l | v
+----+------------------------------------------------------+
0 | !__aaawwmqmqmwwwaas,,_ .__aaawwwmqmqmwwaaa,,
2 | !"VT?!"""^~~^"""??T$Wmqaa,_auqmWBT?!"""^~~^^""??YV^
4 | ! "?##mW##?"-
6 | ! C O N G R A T S _am#Z??A#ma, Y
8 | ! _ummY" "9#ma, A
10 | ! vm#Z( )Xmms Y
12 | ! .j####mmm#####mm#m##6.
14 | ! W O W ! jmm###mm######m#mmm##6
16 | ! ]#me*Xm#m#mm##m#m##SX##c
18 | ! dm#||+*$##m#mm#m#Svvn##m
20 | ! :mmE=|+||S##m##m#1nvnnX##; A
22 | ! :m#h+|+++=Xmm#m#1nvnnvdmm; M
24 | ! Y $#m>+|+|||##m#1nvnnnnmm# A
26 | ! O ]##z+|+|+|3#mEnnnnvnd##f Z
28 | ! U D 4##c|+|+|]m#kvnvnno##P E
30 | ! I 4#ma+|++]mmhvnnvq##P` !
32 | ! D I ?$#q%+|dmmmvnnm##!
34 | ! T -4##wu#mm#pw##7'
36 | ! -?$##m####Y'
38 | ! !! "Y##Y"-
40 | !
(21 rows)
```
4. Exit the SQL shell:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> \q
```
## Step 5. Verify replication
1. To understand replication in CockroachDB, it's important to review a few concepts from the architecture:
| Concept | Description |
| - | - |
| **Range** | CockroachDB stores all user data (tables, indexes, etc.) and almost all system data in a giant sorted map of key-value pairs. This keyspace is divided into "ranges", contiguous chunks of the keyspace, so that every key can always be found in a single range.
From a SQL perspective, a table and its secondary indexes initially map to a single range, where each key-value pair in the range represents a single row in the table (also called the primary index because the table is sorted by the primary key) or a single row in a secondary index. As soon as that range reaches the maximum range size, it splits into two ranges. This process continues for these new ranges as the table and its indexes continue growing. |
| **Replica** | CockroachDB replicates each range (3 times by default) and stores each replica on a different node. |
2. With those concepts in mind, open the DB Console at [http://localhost:8080](http://localhost:8080) and log in with the `maxroach` user.
3. On the **Overview** page, note that the **Replicas** count is the same on all three nodes. This indicates:
* There are this many "ranges" of data in the cluster. These are mostly internal "system" ranges since you haven't added much table data.
* Each range has been replicated 3 times (according to the CockroachDB default).
* For each range, each replica is stored on different nodes.
## Step 6. Add two more nodes
1. Back in the terminal, add a fourth node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--store=rep-node4 \
--listen-addr=localhost:26260 \
--http-addr=localhost:8083 \
--join=localhost:26257,localhost:26258,localhost:26259
```
2. In a new terminal, add a fifth node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--store=rep-node5 \
--listen-addr=localhost:26261 \
--http-addr=localhost:8084 \
--join=localhost:26257,localhost:26258,localhost:26259
```
## Step 7. Watch data rebalance
Back in the DB Console, you'll see that there are now 5 nodes listed.
At first, the replica count will be lower for nodes 4 and 5. Very soon, however, you'll see those numbers even out across all nodes, indicating that data is being automatically rebalanced to utilize the additional capacity of the new nodes.
## Step 8. Stop the cluster
1. When you're done with your test cluster, press `ctrl-c` in each terminal where a node is running.
For the last 2 nodes, the shutdown process will take longer (about a minute each) and will eventually force the nodes to stop. This is because, with only 2 of 5 nodes left, a majority of replicas are not available, and so the cluster is no longer operational. To speed up this process, you can press `ctrl-c` a second time.
1. To restart the cluster at a later time, run the same `cockroach start` commands as earlier from the directory containing the nodes' data stores.
If you do not plan to restart the cluster, you may want to remove the cluster's certificates and data stores:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ rm -rf certs my-safe-directory rep-node1 rep-node2 rep-node3 rep-node4 rep-node5
```
## What's next?
Explore other CockroachDB benefits and features:
* Replication & Rebalancing
* CockroachDB Resilience
* Serializable Transactions
# Serializable Transactions
Source: https://docs.cockroachlabs.com/docs/v26.3/demo-serializable
In contrast to most databases, CockroachDB offers `SERIALIZABLE` isolation by default, which is the strongest of the four [transaction isolation levels](https://wikipedia.org/wiki/Isolation_\(database_systems\)) defined by the SQL standard and is stronger than the `SNAPSHOT` isolation level developed later. `SERIALIZABLE` isolation guarantees that even though transactions may execute in parallel, the result is the same as if they had executed one at a time, without any concurrency. This ensures data correctness by preventing all "anomalies" allowed by weaker isolation levels.
In this tutorial, you'll work through a hypothetical scenario that demonstrates the effectiveness of `SERIALIZABLE` isolation for maintaining data correctness.
1. You'll start by reviewing the scenario and its schema.
2. You'll finish by executing the scenario at `SERIALIZABLE` isolation, observing how it guarantees correctness. You'll use CockroachDB for this portion.
For a deeper discussion of transaction isolation and the write skew anomaly, see the [Real Transactions are Serializable](https://www.cockroachlabs.com/blog/acid-rain/) and [What Write Skew Looks Like](https://www.cockroachlabs.com/blog/what-write-skew-looks-like/) blog posts.
## Overview
### Scenario
* A hospital has an application for doctors to manage their on-call shifts.
* The hospital has a rule that at least one doctor must be on call at any one time.
* Two doctors are on-call for a particular shift, and both of them try to request leave for the shift at approximately the same time.
* In PostgreSQL, with the default `READ COMMITTED` isolation level, the [write skew](#write-skew) anomaly results in both doctors successfully booking leave and the hospital having no doctors on call for that particular shift.
* In CockroachDB, with the `SERIALIZABLE` isolation level, write skew is prevented, one doctor is allowed to book leave and the other is left on-call, and lives are saved.
#### Write skew
When write skew happens, a transaction reads something, makes a decision based on the value it saw, and writes the decision to the database. However, by the time the write is made, the premise of the decision is no longer true. Only `SERIALIZABLE` and some implementations of `REPEATABLE READ` isolation prevent this anomaly.
### Schema
![Schema for serializable transaction tutorial]()
## Step 1. Set up the scenario on PostgreSQL
1. If you haven't already, install PostgreSQL locally. On Mac, you can use [Homebrew](https://brew.sh/):
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ brew install postgres
```
2. [Start PostgreSQL](https://www.postgresql.org/docs/10/static/server-start.html):
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ postgres -D /usr/local/var/postgres &
```
3. Open a SQL connection to PostgreSQL:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ psql
```
4. Create the `doctors` table:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> CREATE TABLE doctors (
id INT PRIMARY KEY,
name TEXT
);
```
5. Create the `schedules` table:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> CREATE TABLE schedules (
day DATE,
doctor_id INT REFERENCES doctors (id),
on_call BOOL,
PRIMARY KEY (day, doctor_id)
);
```
6. Add two doctors to the `doctors` table:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> INSERT INTO doctors VALUES
(1, 'Abe'),
(2, 'Betty');
```
7. Insert one week's worth of data into the `schedules` table:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> INSERT INTO schedules VALUES
('2024-10-01', 1, true),
('2024-10-01', 2, true),
('2024-10-02', 1, true),
('2024-10-02', 2, true),
('2024-10-03', 1, true),
('2024-10-03', 2, true),
('2024-10-04', 1, true),
('2024-10-04', 2, true),
('2024-10-05', 1, true),
('2024-10-05', 2, true),
('2024-10-06', 1, true),
('2024-10-06', 2, true),
('2024-10-07', 1, true),
('2024-10-07', 2, true);
```
8. Confirm that at least one doctor is on call each day of the week:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SELECT day, count(*) AS doctors_on_call FROM schedules
WHERE on_call = true
GROUP BY day
ORDER BY day;
```
```
day | doctors_on_call
------------+-----------------
2024-10-01 | 2
2024-10-02 | 2
2024-10-03 | 2
2024-10-04 | 2
2024-10-05 | 2
2024-10-06 | 2
2024-10-07 | 2
(7 rows)
```
## Step 2. Run the scenario on PostgreSQL
1. Doctor 1, Abe, starts to request leave for 10/5/18 using the hospital's schedule management application. The application starts a transaction:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> BEGIN;
```
2. The application checks to make sure at least one other doctor is on call for the requested date:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SELECT count(*) FROM schedules
WHERE on_call = true
AND day = '2024-10-05'
AND doctor_id != 1;
```
```
count
-------
1
(1 row)
```
3. Around the same time, doctor 2, Betty, starts to request leave for the same day using the hospital's schedule management application. In a new terminal, start a second SQL session:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ psql
```
4. The application starts a transaction:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> BEGIN;
```
5. The application checks to make sure at least one other doctor is on call for the requested date:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SELECT count(*) FROM schedules
WHERE on_call = true
AND day = '2024-10-05'
AND doctor_id != 2;
```
```
count
-------
1
(1 row)
```
6. In the terminal for doctor 1, since the previous check confirmed that another doctor is on call for 10/5/18, the application tries to update doctor 1's schedule:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> UPDATE schedules SET on_call = false
WHERE day = '2024-10-05'
AND doctor_id = 1;
```
7. In the terminal for doctor 2, since the previous check confirmed the same thing, the application tries to update doctor 2's schedule:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> UPDATE schedules SET on_call = false
WHERE day = '2024-10-05'
AND doctor_id = 2;
```
8. In the terminal for doctor 1, the application commits the transaction, despite the fact that the previous check (the `SELECT` query) is no longer true:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> COMMIT;
```
9. In the terminal for doctor 2, the application commits the transaction, despite the fact that the previous check (the `SELECT` query) is no longer true:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> COMMIT;
```
## Step 3. Check data correctness on PostgreSQL
So what just happened? Each transaction started by reading a value that, before the end of the transaction, became incorrect. Despite that fact, each transaction was allowed to commit. This is known as write skew, and the result is that 0 doctors are scheduled to be on call on 10/5/18.
To check this, in either terminal, run:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SELECT * FROM schedules WHERE day = '2024-10-05';
```
```
day | doctor_id | on_call
------------+-----------+---------
2024-10-05 | 1 | f
2024-10-05 | 2 | f
(2 rows)
```
Again, this anomaly is the result of PostgreSQL's default isolation level of `READ COMMITTED`, but note that this would happen with any isolation level except `SERIALIZABLE` and some implementations of `REPEATABLE READ`:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SHOW TRANSACTION_ISOLATION;
```
```
transaction_isolation
-----------------------
read committed
(1 row)
```
Exit each SQL shell with `\q` and then stop the PostgreSQL server:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ pkill -9 postgres
```
## Step 4. Set up the scenario on CockroachDB
When you repeat the scenario on CockroachDB, you'll see that the anomaly is prevented by CockroachDB's `SERIALIZABLE` transaction isolation.
1. If you haven't already, install CockroachDB locally.
2. Use the `cockroach start-single-node` command to start a one-node CockroachDB cluster in insecure mode:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start-single-node \
--insecure \
--store=serializable-demo \
--listen-addr=localhost
```
3. In a new terminal window, open the built-in SQL client and connect to `localhost`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --insecure --host=localhost
```
4. Create the `doctors` table:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> CREATE TABLE doctors (
id INT PRIMARY KEY,
name TEXT
);
```
5. Create the `schedules` table:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> CREATE TABLE schedules (
day DATE,
doctor_id INT REFERENCES doctors (id),
on_call BOOL,
PRIMARY KEY (day, doctor_id)
);
```
6. Add two doctors to the `doctors` table:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> INSERT INTO doctors VALUES
(1, 'Abe'),
(2, 'Betty');
```
7. Insert one week's worth of data into the `schedules` table:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> INSERT INTO schedules VALUES
('2024-10-01', 1, true),
('2024-10-01', 2, true),
('2024-10-02', 1, true),
('2024-10-02', 2, true),
('2024-10-03', 1, true),
('2024-10-03', 2, true),
('2024-10-04', 1, true),
('2024-10-04', 2, true),
('2024-10-05', 1, true),
('2024-10-05', 2, true),
('2024-10-06', 1, true),
('2024-10-06', 2, true),
('2024-10-07', 1, true),
('2024-10-07', 2, true);
```
8. Confirm that at least one doctor is on call each day of the week:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SELECT day, count(*) AS on_call FROM schedules
WHERE on_call = true
GROUP BY day
ORDER BY day;
```
```
day | on_call
-------------+----------
2024-10-01 | 2
2024-10-02 | 2
2024-10-03 | 2
2024-10-04 | 2
2024-10-05 | 2
2024-10-06 | 2
2024-10-07 | 2
(7 rows)
```
## Step 5. Run the scenario on CockroachDB
1. Doctor 1, Abe, starts to request leave for 10/5/18 using the hospital's schedule management application. The application starts a transaction:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> BEGIN;
```
2. The application checks to make sure at least one other doctor is on call for the requested date:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SELECT count(*) FROM schedules
WHERE on_call = true
AND day = '2024-10-05'
AND doctor_id != 1;
```
```
count
---------
1
(1 row)
```
3. Around the same time, doctor 2, Betty, starts to request leave for the same day using the hospital's schedule management application. In a new terminal, start a second SQL session:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --insecure --host=localhost
```
4. The application starts a transaction:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> BEGIN;
```
5. The application checks to make sure at least one other doctor is on call for the requested date:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SELECT count(*) FROM schedules
WHERE on_call = true
AND day = '2024-10-05'
AND doctor_id != 2;
```
```
count
---------
1
(1 row)
```
6. In the terminal for doctor 1, since the previous check confirmed that another doctor is on call for 10/5/18, the application tries to update doctor 1's schedule:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> UPDATE schedules SET on_call = false
WHERE day = '2024-10-05'
AND doctor_id = 1;
```
7. In the terminal for doctor 2, since the previous check confirmed the same thing, the application tries to update doctor 2's schedule:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> UPDATE schedules SET on_call = false
WHERE day = '2024-10-05'
AND doctor_id = 2;
```
8. In the terminal for doctor 1, the application tries to commit the transaction:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> COMMIT;
```
The transaction for doctor 1 is committed.
9. In the terminal for doctor 2, the application tries to commit the transaction:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> COMMIT;
```
Since CockroachDB uses `SERIALIZABLE` isolation, the database detects that the previous check (the `SELECT` query) is no longer true due to a concurrent transaction. It therefore prevents the transaction from committing, returning a retry error that indicates that the transaction must be attempted again.
```
ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to encountered recently written committed value /Table/105/1/20001/1/0 @1700513356.063385000,2): "sql txn" meta={id=10f4abbc key=/Table/105/1/20001/2/0 iso=Serializable pri=0.00167708 epo=0 ts=1700513366.194063000,2 min=1700513327.262632000,0 seq=1} lock=true stat=PENDING rts=1700513327.262632000,0 wto=false gul=1700513327.762632000,0
SQLSTATE: 40001
HINT: See: https://www.cockroachlabs.com/docs/v26.3/transaction-retry-error-reference.html#retry_serializable
```
For this kind of error, CockroachDB recommends a client-side transaction retry loop that would transparently observe that the one doctor cannot take time off because the other doctor already succeeded in asking for it. You can find generic transaction retry functions for various languages in the Develop with CockroachDB section.
For more information about the error message for the `RETRY_SERIALIZABLE` error type, see the Transaction Retry Error Reference.
## Step 6. Check data correctness on CockroachDB
1. In either terminal, confirm that one doctor is still on call for 10/5/18:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SELECT * FROM schedules WHERE day = '2024-10-05';
```
```
day | doctor_id | on_call
-------------+-----------+----------
2024-10-05 | 1 | f
2024-10-05 | 2 | t
(2 rows)
```
2. Again, the write skew anomaly was prevented by CockroachDB using the `SERIALIZABLE` isolation level:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SHOW TRANSACTION_ISOLATION;
```
```
transaction_isolation
-------------------------
serializable
(1 row)
```
3. Exit the SQL shell in each terminal:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> \q
```
4. Exit each SQL shell with `\q` and then stop the node:
Get the process ID of the node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
ps -ef | grep cockroach | grep -v grep
```
```
501 21691 1 0 6:19PM ttys001 0:01.15 cockroach start-single-node --insecure --store=serializable-demo --listen-addr=localhost
```
Gracefully shut down the node, specifying its process ID:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kill -TERM 21691
```
```
initiating graceful shutdown of server
server drained and shutdown completed
```
If you do not plan to restart the cluster, you may want to remove the node's data store:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ rm -rf serializable-demo
```
## What's next?
Explore other CockroachDB benefits and features:
* Replication & Rebalancing
* CockroachDB Resilience
* Serializable Transactions
You might also want to learn more about how transactions work in CockroachDB and in general:
* Transactions Overview
* [What Write Skew Looks Like](https://www.cockroachlabs.com/blog/what-write-skew-looks-like/)
* Read Committed Transactions
# Deploy a CockroachDB Cloud Application with Google Cloud Run
Source: https://docs.cockroachlabs.com/docs/v26.3/deploy-app-gcr
This tutorial shows you how to use Google Cloud Run to deploy a containerized Django application that communicates with a CockroachDB Standard cluster.
## Before you begin
Before starting the tutorial, do the following:
1. Create a [Google Cloud](https://cloud.google.com/) account and log in.
2. Install the [Google Cloud SDK](https://cloud.google.com/sdk).
3. Install [Docker Desktop](https://www.docker.com/products/docker-desktop).
## Step 1. Create a CockroachDB Standard cluster
1. Create a CockroachDB Cloud account. If this is your first CockroachDB Cloud organization, it will be credited with \$400 in free trial credits to get you started.
2. On the **Get Started** page, click **Create cluster**.
3. On the **Select a plan** page, select **Standard**.
4. On the **Cloud & Regions** page, select a cloud provider (GCP or AWS).
5. In the **Regions** section, select a region for the cluster. Refer to CockroachDB Cloud Regions for the regions where CockroachDB Standard clusters can be deployed. To create a multi-region cluster, click **Add region** and select additional regions.
6. Click **Next: Capacity**.
7. On the **Capacity** page, keep the **Provisioned capacity** at the default value of 2 vCPUs.
Click **Next: Finalize**.
8. On the **Finalize** page, name your cluster. If an active free trial is listed in the right pane, you will not need to add a payment method, though you will need to do this by the end of the trial to maintain your organization's clusters.
Click **Create cluster**.
Your cluster will be created in a few seconds and the **Create SQL user** dialog will display.
## Step 2. Set up your cluster connection
Once your cluster is created, the **Connect to cluster-name** dialog displays. Use the information provided in the dialog to set up your cluster connection for the SQL user that was created by default:
1. In your terminal, run the second command from the dialog to create a new `certs` directory on your local machine and download the CA certificate to that directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl --create-dirs -o ~/.postgresql/root.crt -O https://cockroachlabs.cloud/clusters/{cluster-id}/cert
```
Your `cert` file will be downloaded to `~/.postgresql/root.crt`.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl --create-dirs -o ~/.postgresql/root.crt -O https://cockroachlabs.cloud/clusters/{cluster-id}/cert
```
Your `cert` file will be downloaded to `~/.postgresql/root.crt`.
```powershell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
mkdir -p $env:appdata\.postgresql\; Invoke-WebRequest -Uri https://cockroachlabs.cloud/clusters/{cluster-id}/cert -OutFile $env:appdata\.postgresql\root.crt
```
Your `cert` file will be downloaded to `%APPDATA%/.postgresql/root.crt`.
2. Copy the connection string provided, which will be used in the next steps (and to connect to your cluster in the future).
This connection string contains your password, which will be provided only once. If you forget your password, you can reset it by going to the **SQL Users** page for the cluster, found at `https://cockroachlabs.cloud/cluster//users`.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroach sql --url 'postgresql://download the CockroachDB SQL Shell binary.
2. Start the built-in SQL shell using the connection string you got from the CockroachDB Cloud Console earlier:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroach sql \
--url='postgres:// CREATE DATABASE bank;
```
4. Exit the SQL shell:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> \q
```
## Step 4. Get the application code
1. Clone the example code's GitHub repo:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
git clone https://github.com/cockroachlabs/example-app-python-django/
```
2. Create a new directory named `certs` at the top level of the `example-app-python-django` project, and then copy the root certificate that you downloaded for your cluster to the new directory.
The project directory structure should look like this:
```
├── Dockerfile
├── README.md
├── certs
│ └── root.crt
├── cockroach_example
│ ├── cockroach_example
│ │ ├── __init__.py
│ │ ├── asgi.py
│ │ ├── migrations
│ │ │ ├── 0001_initial.py
│ │ │ └── __init__.py
│ │ ├── models.py
│ │ ├── settings.py
│ │ ├── urls.py
│ │ ├── views.py
│ │ └── wsgi.py
│ └── manage.py
└── requirements.txt
```
## Step 5. Initialize the database and test the app locally
1. At the top level of the app's project directory, create and then activate a virtual environment:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
virtualenv env
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
source env/bin/activate
```
2. Install the required modules to the virtual environment:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
pip install -r requirements.txt
```
3. Set the `DATABASE_URL` environment variable to the connection string provided in the **Connection info** window of the CockroachDB Cloud Console, but with the root certificate located in the local `certs` directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="postgresql://$USER:$PASSWORD@random-cluster-name-4300.6wr.cockroachlabs.cloud:26257/defaultdb?sslmode=verify-full&sslrootcert=root.crt"
```
This Django app uses the `dj_database_url` module to configure the database connection from a connection URL. The module uses the value assigned to the `DATABASE_URL` environment variable for the connection.
In the Cloud Run deployment, we use the Google Cloud Secret Manager to define the `DATABASE_URL` environment variable for the deployment.
1. Execute the initial database schema migration:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
python3 cockroach_example/manage.py migrate
```
This migration initializes the `bank` database with the tables defined in `models.py`, in addition to some other tables for the admin functionality included with Django's starter application.
2. Run the app:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
python3 cockroach_example/manage.py runserver 0.0.0.0:8000
```
The output should look like this:
```
...
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.
```
To perform simple reads and writes to the database, you can send HTTP requests to the application at [http://0.0.0.0:8000/](http://0.0.0.0:8000/).
3. In a new terminal, use `curl` to send a POST request to the application:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl --header "Content-Type: application/json" \
--request POST \
--data '{"name":"Carl"}' http://0.0.0.0:8000/customer/
```
This request inserts a new row into the `cockroach_example_customers` table.
4. Send a GET request to read from the `cockroach_example_customers` table:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl http://0.0.0.0:8000/customer/
```
```
[{"id": "bb7d6c4d-efb3-45f8-b790-9911aae7d8b2", "name": "Carl"}]
```
You can also query the table directly in the SQL shell to see the changes:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SELECT * FROM bank.cockroach_example_customers;
```
```
id | name
---------------------------------------+-------
bb7d6c4d-efb3-45f8-b790-9911aae7d8b2 | Carl
(1 row)
```
5. Enter **Ctrl+C** to stop the application.
## Step 6. Configure GCP
1. In the terminal, authenticate the `gcloud` command-line tool with your Google Cloud account:
`gcloud` is included with the [Google Cloud SDK](https://cloud.google.com/sdk) installation.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
gcloud auth login
```
Follow the prompts to authenticate.
1. Create a Google Cloud project for the application deployment:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
gcloud projects create {gcp_project_id}
```
You can specify a location for the project within your Google Cloud resources by using the `--organization` or `--folder` flags.
1. Configure the CLI to use your Google Cloud account and the new project ID by default:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
gcloud init
```
2. Set the `PROJECT_ID` environment variable:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export PROJECT_ID={gcp_project_id}
```
For the rest of the tutorial, we use `PROJECT_ID` to refer to the project ID.
## Step 7. Containerize the application and push it to the registry
1. Build the Docker image locally:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
docker build -t gcr.io/$PROJECT_ID/crdb-sample:v1 .
```
If there are no errors, the container built successfully.
2. Authenticate Docker with GCP's Container Registry:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
gcloud auth configure-docker
```
3. Enable the Container Registry API for the project:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
gcloud services enable containerregistry.googleapis.com
```
4. Push the Docker image to the project's registry.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
docker push gcr.io/$PROJECT_ID/crdb-sample:v1
```
## Step 8. Create a secret for the database connection URI
1. Create a service account to manage the secrets for your project:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
gcloud iam service-accounts create cockroach-labs
```
2. Enable the Secret Manager API for the project:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
gcloud services enable secretmanager.googleapis.com
```
3. Create a secret for the connection string stored locally in the `DATABASE_URL` environment variable, and bind the new service account to the secret.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
echo $DATABASE_URL | gcloud secrets create cockroach-connection-uri --data-file=- --replication-policy=automatic
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
gcloud secrets add-iam-policy-binding cockroach-connection-uri \
--member=serviceAccount:cockroach-labs@${PROJECT_ID}.iam.gserviceaccount.com \
--role=roles/secretmanager.secretAccessor
```
## Step 9. Deploy the application on Cloud Run
1. Enable the Cloud Run API for the project:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
gcloud services enable run.googleapis.com
```
2. Create a [Cloud Run](https://console.cloud.google.com/run/) service for the application:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
gcloud alpha run deploy crl-app \
--region us-central1 \
--allow-unauthenticated \
--service-account "cockroach-labs@${PROJECT_ID}.iam.gserviceaccount.com" \
--set-secrets "DATABASE_URL=cockroach-connection-uri:latest" \
--image gcr.io/${PROJECT_ID}/crdb-sample:v1
```
Note the following:
* The `--region` flag specifies the region of the CockroachDB node targeted in the connection string.
* The `--service-account` flag specifies the `cockroach-labs` service account that you created earlier for the app deployment.
* The `--set-secrets` flag sets the `DATABASE_URL` environment variable to the `cockroach-connection-uri` secret that you created earlier.
* The `--image` flag specifies the container image URL for the `crdb-sample` image that you pushed to the container registry.
If prompted, select `Cloud Run (fully managed)`.
3. After the revision is deployed, verify that you can send requests to the application from a browser or a REST client:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl https:///customer/
```
```
[{"id": "bb7d6c4d-efb3-45f8-b790-9911aae7d8b2", "name": "Carl"}]
```
4. By default, the sample application allows all hosts/domain names to serve the application. After testing, we recommend that you update the [`ALLOWED_HOSTS` property in `settings.py`](https://docs.djangoproject.com/en/3.2/ref/settings/#allowed-hosts) to allow only a local testing URL and the Cloud Run service URL to serve the application.
## See also
* Build a Simple Django App with CockroachDB
You might also be interested in the following pages:
* Client Connection Parameters
* Connection Pooling
* Data Replication
* CockroachDB Resilience
* Replication & Rebalancing
# Deploy a Netlify App Built on CockroachDB
Source: https://docs.cockroachlabs.com/docs/v26.3/deploy-app-netlify
This tutorial shows you how to deploy a [Netlify](https://www.netlify.com/) web application that communicates with a CockroachDB Standard cluster.
The sample app used in this tutorial simulates a [gaming leaderboard](https://www.cockroachlabs.com/blog/react-typescript-cockroachdb-sample-app/) using [Netlify functions](https://www.netlify.com/products/functions/) written in TypeScript. The functions use [Prisma](https://www.prisma.io/) to connect to CockroachDB. The app's frontend, also written in TypeScript, uses React, bootstrapped with [Create React App](https://github.com/facebook/create-react-app).
You can view or download the [source code](https://github.com/cockroachdb/cockroachdb-typescript) for the example.
## Before you begin
Before starting the tutorial:
1. Create a Starter [Netlify](https://app.netlify.com/signup) account.
2. Install the `netlify` CLI locally and log in. Refer to [Get started with Netlify CLI](https://docs.netlify.com/cli/get-started/).
## Step 1. Create a CockroachDB Standard cluster
1. Create a CockroachDB Cloud account. If this is your first CockroachDB Cloud organization, it will be credited with \$400 in free trial credits to get you started.
2. On the **Get Started** page, click **Create cluster**.
3. On the **Select a plan** page, select **Standard**.
4. On the **Cloud & Regions** page, select a cloud provider (GCP or AWS).
5. In the **Regions** section, select a region for the cluster. Refer to CockroachDB Cloud Regions for the regions where CockroachDB Standard clusters can be deployed. To create a multi-region cluster, click **Add region** and select additional regions.
6. Click **Next: Capacity**.
7. On the **Capacity** page, keep the **Provisioned capacity** at the default value of 2 vCPUs.
Click **Next: Finalize**.
8. On the **Finalize** page, name your cluster. If an active free trial is listed in the right pane, you will not need to add a payment method, though you will need to do this by the end of the trial to maintain your organization's clusters.
Click **Create cluster**.
Your cluster will be created in a few seconds and the **Create SQL user** dialog will display.
After the cluster is created, the **Connection info** window appears. Click the **Connection string** tab and copy the connection string to a secure location. You will use this connection string to connect to CockroachDB later in the tutorial.
The connection string is pre-populated with your SQL username and password, the cluster name, and other details. Save the password in a secure place such a password manager. You can find the connection string at any time from the CockroachDB Cloud Console. You can reset your SQL password for this cluster from the cluster's **SQL Users** page.
## Step 2. Get the code
1. Open the code repository's [GitHub repository](https://github.com/cockroachdb/cockroachdb-typescript).
2. Fork it by clicking **Fork** at the top. Netlify requires you to own a Netlify app's repository.
* Set **Owner** to your GitHub identity or an organization.
* Disable **Copy the master branch only**.
* Click **Fork**.
You now have an exact copy of the code repository in the GitHub location you chose.
3. Copy the fork's address, which you will use to clone it locally. Click **Code**, then in the **Local** tab, click **SSH** if you have added SSH keys to GitHub, or **HTTPS** if not. Click the copy symbol to copy the address.
4. Clone your fork locally. Replace with the full address of your fork, which will end with `cockroachdb-typescript.git`.
The project has the following directory structure:
```
├── LICENSE.md
├── README.md
├── netlify
│ └── functions
│ ├── addScore.ts
│ ├── getPlayers.ts
│ └── getScores.ts
├── package-lock.json
├── package.json
├── prisma
│ ├── schema.prisma
│ └── seed.ts
├── public
├── src
└── tsconfig.json
```
This tutorial modifies the files in the `netlify` and `prisma` directories.
## Step 3. Initialize the database
1. Open the project's `.env` file in a text editor set the `DATABASE_URL` environment variable to [the connection string](#connection-string) you obtained earlier from the CockroachDB Cloud Console:
```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
DATABASE_URL=`SELECT` and `INSERT` queries against the database:
* [`getScores.ts`](https://raw.githubusercontent.com/cockroachdb/cockroachdb-typescript/master/netlify/functions/getScores.ts) reads all rows from the `player_scores` table and returns values in the `id`, `name`, and `score` columns.
* [`getPlayers.ts`](https://raw.githubusercontent.com/cockroachdb/cockroachdb-typescript/master/netlify/functions/getPlayers.ts) reads and returns all rows from the `players` table.
* [`addScore.ts`](https://raw.githubusercontent.com/cockroachdb/cockroachdb-typescript/master/netlify/functions/addScore.ts) writes new scores to the `player_scores` table.
2. Deploy your app with the Netlify CLI:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
netlify deploy
```
Choose to create a new site, and then select the default options for each of the subsequent prompts. You will be required to log in to Netlify using GitHub.
After the app is deployed, you should see the following message:
```
✔ Deploy is live!
```
3. Navigate to the admin URL for your site:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
netlify open
```
From the **Site overview** page, you can manage your site. The **Site overview** page also provides you with a public URL for your site.
## See also
* [How to build a Complete Webapp with React, TypeScript & CockroachDB](https://www.cockroachlabs.com/blog/react-typescript-cockroachdb-sample-app/#deploy-the-application-to-netlify)
* Build a Simple CRUD Node.js App with CockroachDB and Prisma Client
You might also be interested in the following pages:
* Client Connection Parameters
* Connection Pooling
* Data Replication
* CockroachDB Resilience
* Replication & Rebalancing
# Deploy a Web App Built on CockroachDB with Vercel
Source: https://docs.cockroachlabs.com/docs/v26.3/deploy-app-vercel
This tutorial shows you how to use [Vercel](https://vercel.com/) to deploy a web application built with [Next.js](https://nextjs.org/), [Express](https://expressjs.com/), [Prisma](https://www.prisma.io/), and CockroachDB Standard.
## Before you begin
Before starting the tutorial, create a [Vercel](https://vercel.com/signup) account and sign in.
## Step 1. Create a CockroachDB Standard cluster
1. Create a CockroachDB Cloud account. If this is your first CockroachDB Cloud organization, it will be credited with \$400 in free trial credits to get you started.
2. On the **Get Started** page, click **Create cluster**.
3. On the **Select a plan** page, select **Standard**.
4. On the **Cloud & Regions** page, select a cloud provider (GCP or AWS).
5. In the **Regions** section, select a region for the cluster. Refer to CockroachDB Cloud Regions for the regions where CockroachDB Standard clusters can be deployed. To create a multi-region cluster, click **Add region** and select additional regions.
6. Click **Next: Capacity**.
7. On the **Capacity** page, keep the **Provisioned capacity** at the default value of 2 vCPUs.
Click **Next: Finalize**.
8. On the **Finalize** page, name your cluster. If an active free trial is listed in the right pane, you will not need to add a payment method, though you will need to do this by the end of the trial to maintain your organization's clusters.
Click **Create cluster**.
Your cluster will be created in a few seconds and the **Create SQL user** dialog will display.
After the cluster is created, the **Connection info** window appears. Click the **Connection string** tab and copy the connection string to a secure location. You will use this connection string to connect to CockroachDB later in the tutorial.
The connection string is pre-populated with your SQL username and password, the cluster name, and other details. Save the password in a secure place such a password manager. You can find the connection string at any time from the CockroachDB Cloud Console. You can reset your SQL password for this cluster from the cluster's **SQL Users** page.
## Step 2. Get the code
1. Open the code repository's [GitHub repository](https://github.com/cockroachdb/prisma-examples).
2. Fork it by clicking **Fork** at the top. Netlify requires you to own a Netlify app's repository.
* Set **Owner** to your GitHub identity or an organization.
* Disable **Copy the master branch only**.
* Click **Fork**.
You now have an exact copy of the code repository in the GitHub location you chose.
3. Copy the fork's address, which you will use to clone it locally. Click **Code**, then in the **Local** tab, click **SSH** if you have added SSH keys to GitHub, or **HTTPS** if not. Click the copy symbol to copy the address.
4. Clone your fork locally. Replace with the full address of your fork, which will end with `prisma-examples.git`.
## Step 3. Create a Vercel project
1. From the [Vercel dashboard](https://vercel.com/dashboard), click **Add new\...** > **Project**.
2. Select the `prisma-examples` repository you forked in [Step 2](#step-2-get-the-code) to import.
3. From the **Framework preset** dropdown, select **Next.js**.
4. In the **Root directory** field, click **Edit**.
5. Select the **typescript** > **rest-nextjs-api-routes** directory.
6. Click **Continue**.
7. Open the **Build and Output Settings** and toggle the build command's **Override** switch to on.
8. Enter `yarn prisma db push && yarn next build` as the **Build command**.
9. Click **Deploy**.
Your deployment will fail until you integrate CockroachDB in the next step, so you can leave this screen without waiting for it to finish.
## Step 4. Integrate your project with CockroachDB
1. Navigate to the [CockroachDB page](https://vercel.com/integrations/cockroachdb) of Vercel's integration marketplace.
2. Click **Add Integration**.
3. Select your Vercel account and click **Continue**.
4. Select the project you just created and click **Continue**.
5. Accept the permissions and click **Add Integration**.
6. When prompted, log into CockroachDB Cloud and select the organization and cluster to integrate with.
7. Click **Create** to finish the integration.
After a few seconds, the window will close automatically and your Vercel project will have the `DATABASE_URL` environment variable configured with the cluster's connection string.
## Step 5. Deploy the application
1. Navigate to the **Overview** page for your Vercel project.
2. Select the **Deployments** tab.
3. Click **Redeploy**.
4. Click **Redeploy** again from the pop-up dialog.
Your project will be deployed in a few seconds. You can view the final product by clicking on the **Domain** link on your project's **Overview** page.
## Step 3. Initialize the database
1. Save [the connection string](#connection-string) you obtained earlier from the CockroachDB Cloud Console to the `DATABASE_URL` environment variable in an `.env` file in your project:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
echo "DATABASE_URL=> .env
```
Prisma loads the variables defined in `.env` to the project environment. By default, Prisma uses the `DATABASE_URL` environment variable as the connection string to the database.
2. Run [Prisma Migrate](https://www.prisma.io/docs/concepts/components/prisma-migrate) to initialize the database with the schema defined in `prisma/prisma.schema`.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
prisma migrate dev --name init
```
You should see the following output:
```
Your database is now in sync with your schema.
✔ Generated Prisma Client (3.11.0 | library) to ./node_modules/@prisma/client in 87ms
```
This command also initializes [Prisma Client](https://www.prisma.io/docs/concepts/components/prisma-client) to communicate with your CockroachDB cluster, based on the configuration in the `prisma/schema.prisma` file.
## Step 4. Deploy the application
1. (Optional) Run the app server locally to preview your site:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
npm run dev
```
You should see the following output:
```
ready - started server on 0.0.0.0:3000, url: http://localhost:3000
```
2. Run the `vercel` command to start deploying the app:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
vercel deploy --confirm
```
You will be asked to sign into your Vercel account. When you authenticate your credentials, you should see the following message:
```
> Success! GitHub authentication complete ...
```
The [`--confirm` flag](https://vercel.com/docs/cli#introduction/unique-options/confirm) instructs Vercel to deploy the app to the `example-app-typescript-vercel` Vercel project, using the default settings:
```
🔗 Linked to username/example-app-typescript-vercel (created .vercel and added it to .gitignore)
🔍 Inspect: https://vercel.com/username/example-app-typescript-vercel/123456789 [1s]
✅ Production: https://example-app-typescript-vercel-username.vercel.app [copied to clipboard] [47s]
📝 Deployed to production. Run `vercel --prod` to overwrite later (https://vercel.link/2F).
💡 To change the domain or build command, go to https://vercel.com/ericharmeling/example-app-typescript-vercel/settings
```
Follow the links provided to view and manage your deployed application.
## See also
* [Vercel's official How to Build a Fullstack App with Next.js, Prisma, and PostgreSQL guide](https://vercel.com/guides/nextjs-prisma-postgres)
* [How to build a Complete Webapp with React, TypeScript & CockroachDB](https://www.cockroachlabs.com/blog/react-typescript-cockroachdb-sample-app/#deploy-the-application-to-netlify)
* Build a Simple CRUD Node.js App with CockroachDB and Prisma Client
You might also be interested in the following pages:
* Client Connection Parameters
* Connection Pooling
* Data Replication
* CockroachDB Resilience
* Replication & Rebalancing
# Deploy CockroachDB on AWS EC2
Source: https://docs.cockroachlabs.com/docs/v26.3/deploy-cockroachdb-on-aws
This page shows you how to manually deploy a multi-node, self-hosted CockroachDB cluster on Amazon's AWS EC2 platform, using AWS's managed load-balancing service to distribute client traffic.
After setting up the AWS network, clock synchronization, and load balancing, it should take approximately 20 minutes to complete the deployment. This is based on initializing a three-node CockroachDB cluster in a single AWS region and running our sample workload.
If you are only testing CockroachDB, or you are not concerned with protecting network communication with TLS encryption, you can use an insecure cluster instead. Select **Insecure** above for instructions.
To try CockroachDB Cloud instead of running CockroachDB yourself, refer to the Cloud Quickstart.
You need a license to use CockroachDB. Refer to the Licensing FAQ and [CockroachDB Pricing](https://www.cockroachlabs.com/pricing). [Contact us](https://cockroachlabs.com/contact-sales) about custom pricing through AWS Marketplace.
## Before you begin
### Requirements
* You must have CockroachDB installed locally. This is necessary for generating and managing your deployment's certificates.
* You must have [SSH access]() to each machine. This is necessary for distributing and starting CockroachDB binaries.
* Your network configuration must allow TCP communication on the following ports:
* `26257` for intra-cluster and client-cluster communication
* `8080` to expose your DB Console
* Carefully review the Production Checklist, including supported hardware and software, and the recommended Topology Patterns.
* Do not run multiple node processes on the same VM or machine. This defeats CockroachDB's replication and causes the system to be a single point of failure. Instead, start each node on a separate VM or machine.
* To start a node with multiple disks or SSDs, provide a separate `--store` flag for each disk when starting the `cockroach` process on the node. For more details about stores, see Start a Node.
If you start a node with multiple `--store` flags, it is not possible to scale back down to only using a single store on the node. Instead, you must decommission the node and start a new node with the updated `--store`.
* When starting each node, use the `--locality` flag to describe the node's location, for example, `--locality=region=west,zone=us-west-1`. The key-value pairs should be ordered from most to least inclusive, and the keys and order of key-value pairs must be the same on all nodes.
* When deploying in a single availability zone:
* To be able to tolerate the failure of any 1 node, use at least 3 nodes with the `default` 3-way replication factor. In this case, if 1 node fails, each range retains 2 of its 3 replicas, a majority.
* To be able to tolerate 2 simultaneous node failures, use at least 5 nodes and increase the `default` replication factor for user data to 5. The replication factor for important internal data is 5 by default, so no adjustments are needed for internal data. In this case, if 2 nodes fail at the same time, each range retains 3 of its 5 replicas, a majority.
* When deploying across multiple availability zones:
* To be able to tolerate the failure of 1 entire AZ in a region, use at least 3 AZs per region and set `--locality` on each node to spread data evenly across regions and AZs. In this case, if 1 AZ goes offline, the 2 remaining AZs retain a majority of replicas.
* To ensure that ranges are split evenly across nodes, use the same number of nodes in each AZ. This is to avoid overloading any nodes with excessive resource consumption.
* When deploying across multiple regions:
* To be able to tolerate the failure of 1 entire region, use at least 3 regions.
CockroachDB is supported in all [AWS regions](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html).
### Recommendations
* Decide how you want to access your DB Console:
| Access Level | Description |
| - | - |
| Partially open | Set a firewall rule to allow only specific IP addresses to communicate on port `8080`. |
| Completely open | Set a firewall rule to allow all IP addresses to communicate on port `8080`. |
| Completely closed | Set a firewall rule to disallow all communication on port `8080`. In this case, a machine with SSH access to a node could use an SSH tunnel to access the DB Console. |
* You should have familiarity with configuring the following AWS components:
* [Amazon VPC](https://docs.aws.amazon.com/vpc/index.html)
* [Subnets](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Subnets.html)
* [Internet gateways](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Internet_Gateway.html)
* [Route tables](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Route_Tables.html)
* [AWS VPN](https://aws.amazon.com/vpn/)
* [Virtual private gateways](https://docs.aws.amazon.com/vpn/latest/s2svpn/VPC_VPN.html)
* [Elastic Load Balancing](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/load-balancer-types.html)
* All Amazon EC2 instances running CockroachDB should be members of the same [security group](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html). For an example, see [AWS architecture](#aws-architecture).
* Follow the [AWS IAM best practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html) to harden the AWS environment. Use roles to grant access to the deployment, following a [policy of least privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege).
* The [AWS root user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_root-user.html) is *not* necessary.
* Review the [AWS Config service limits](https://docs.aws.amazon.com/config/latest/developerguide/configlimits.html) and contact AWS support to request a quota increase, if necessary.
### AWS architecture
In this basic deployment, 3 CockroachDB nodes are each deployed on an Amazon EC2 instance across 3 availability zones. These are grouped within a single VPC and security group. Users are routed to the cluster via [Amazon Route 53](https://aws.amazon.com/route53/) (which is not used in this tutorial) and a load balancer.
![Architecture diagram for a three-node CockroachDB cluster deployed on AWS]()
## Step 1. Create instances
Open the [Amazon EC2 console](https://console.aws.amazon.com/ec2/) and [launch an instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/launching-instance.html#launch-instance-console) for each node you plan to have in your cluster. If you plan to [run our sample workload](#step-9-run-a-sample-workload) against the cluster, create a separate instance for that workload.
* Run at least 3 nodes to ensure survivability.
* Your instances will rely on Amazon Time Sync Service for clock synchronization. When choosing an AMI, note that some machines are preconfigured to use Amazon Time Sync Service (e.g., Amazon Linux AMIs) and others are not.
* Use general-purpose [`m6i` or `m6a`](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/general-purpose-instances.html) VMs with SSD-backed [EBS volumes](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html). For example, Cockroach Labs has used `m6i.2xlarge` for performance benchmarking. If your workload requires high throughput, use network-optimized `m5n` instances. To simulate bare-metal deployments, use `m5d` with [SSD Instance Store volumes](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ssd-instance-store.html).
* `m5` and `m5a` instances, and [compute-optimized `c5`, `c5a`, and `c5n`](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/compute-optimized-instances.html) instances, are also acceptable.
**Do not** use [burstable performance instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-performance-instances.html), which limit the load on a single core.
* Note the ID of the VPC you select. You will need to look up its IP range when setting inbound rules for your security group.
* Make sure all your instances are in the same security group.
* If you are creating a new security group, add the [inbound rules](#step-2-configure-your-network) from the next step. Otherwise note the ID of the security group.
* When creating the instance, you will be prompted to specify an EC2 key pair. For more information on key pairs, see the [AWS documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html). These are used to securely connect to your instances and should be encrypted (e.g., `ssh-keygen -p -f $keypairfile` in Linux).
For more details, see Hardware Recommendations and Cluster Topology.
## Step 2. Configure your network
[Add Custom TCP inbound rules](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html#adding-security-group-rule) to your security group to allow TCP communication on two ports:
* `26257` for inter-node and client-node communication. This enables the nodes to work as a cluster, the load balancer to route traffic to the nodes, and applications to connect to the load balancer.
* `8080` for exposing the DB Console to the user, and for routing the load balancer to the health check endpoint.
#### Inter-node and load balancer-node communication
| Field | Value |
| - | - |
| Port Range | **26257** |
| Source | The ID of your security group (e.g., *sg-07ab277a*) |
#### Application data
| Field | Value |
| - | - |
| Port Range | **26257** |
| Source | Your application's IP ranges |
#### DB Console
| Field | Value |
| - | - |
| Port Range | **8080** |
| Source | Your network's IP ranges |
You can set your network IP by selecting "My IP" in the Source field.
#### Load balancer-health check communication
| Field | Value |
| - | - |
| Port Range | **8080** |
| Source | The IP range of your VPC in CIDR notation (e.g., 10.12.0.0/16) |
To get the IP range of a VPC, open the [Amazon VPC console](https://console.aws.amazon.com/vpc/) and find the VPC listed in the section called Your VPCs.
#### Load balancer-health check communication
| Field | Value |
| - | - |
| Port Range | **8080** |
| Source | The IP range of your VPC in CIDR notation (e.g., 10.12.0.0/16) |
To get the IP range of a VPC, open the [Amazon VPC console](https://console.aws.amazon.com/vpc/) and find the VPC listed in the section called Your VPCs.
## Step 3. Synchronize clocks
CockroachDB requires moderate levels of clock synchronization to preserve data consistency. For this reason, when a node detects that its clock is out of sync with at least half of the other nodes in the cluster by 80% of the maximum offset allowed (500ms by default), it spontaneously shuts down. This avoids the risk of consistency anomalies, but it's best to prevent clocks from drifting too far in the first place by running clock synchronization software on each node.
[`ntpd`](http://doc.ntp.org/) should keep offsets in the single-digit milliseconds, so that software is featured here, but other methods of clock synchronization are suitable as well.
1. SSH to the first machine.
2. Disable `timesyncd`, which tends to be active by default on some Linux distributions:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo timedatectl set-ntp no
```
Verify that `timesyncd` is off:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ timedatectl
```
Look for `Network time on: no` or `NTP enabled: no` in the output.
3. Install the `ntp` package:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo apt-get install ntp
```
4. Stop the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp stop
```
5. Sync the machine's clock with Google's NTP service:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpd -b time.google.com
```
To make this change permanent, in the `/etc/ntp.conf` file, remove or comment out any lines starting with `server` or `pool` and add the following lines:
```
server time1.google.com iburst
server time2.google.com iburst
server time3.google.com iburst
server time4.google.com iburst
```
Restart the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp start
```
We recommend Google's NTP service because it handles ["smearing" the leap second](https://developers.google.com/time/smear). If you use a different NTP service that doesn't smear the leap second, be sure to configure client-side smearing in the same way on each machine. See the Production Checklist for details.
1. Verify that the machine is using a Google NTP server:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpq -p
```
The active NTP server will be marked with an asterisk.
2. Repeat these steps for each machine where a CockroachDB node will run.
Compute Engine instances are preconfigured to use [NTP](http://www.ntp.org/), which should keep offsets in the single-digit milliseconds. However, Google can’t predict how external NTP services, such as `pool.ntp.org`, will handle the leap second. Therefore, you should:
* [Configure each GCE instance to use Google's internal NTP service](https://cloud.google.com/compute/docs/instances/configure-ntp#configure_ntp_for_your_instances).
* If you plan to run a hybrid cluster across GCE and other cloud providers or environments, note that all of the nodes must be synced to the same time source, or to different sources that implement leap second smearing in the same way. See the Production Checklist for details.
Amazon provides the [Amazon Time Sync Service](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html), which uses a fleet of satellite-connected and atomic reference clocks in each AWS Region to deliver accurate current time readings. The service also smears the leap second.
* [Configure each AWS instance to use the internal Amazon Time Sync Service](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html#configure-amazon-time-service).
* Per the above instructions, ensure that `etc/chrony.conf` on the instance contains the line `server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4` and that other `server` or `pool` lines are commented out.
* To verify that Amazon Time Sync Service is being used, run `chronyc sources -v` and check for a line containing `* 169.254.169.123`. The `*` denotes the preferred time server.
* If you plan to run a hybrid cluster across GCE and other cloud providers or environments, note that all of the nodes must be synced to the same time source, or to different sources that implement leap second smearing in the same way. See the Production Checklist for details.
[`ntpd`](http://doc.ntp.org/) should keep offsets in the single-digit milliseconds, so that software is featured here. However, to run `ntpd` properly on Azure VMs, it's necessary to first unbind the Time Synchronization device used by the Hyper-V technology running Azure VMs; this device aims to synchronize time between the VM and its host operating system but has been known to cause problems.
1. SSH to the first machine.
2. Find the ID of the Hyper-V Time Synchronization device:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ curl -O https://raw.githubusercontent.com/torvalds/linux/master/tools/hv/lsvmbus
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ python lsvmbus -vv | grep -w "Time Synchronization" -A 3
```
```
VMBUS ID 12: Class_ID = {9527e630-d0ae-497b-adce-e80ab0175caf} - [Time Synchronization]
Device_ID = {2dd1ce17-079e-403c-b352-a1921ee207ee}
Sysfs path: /sys/bus/vmbus/devices/2dd1ce17-079e-403c-b352-a1921ee207ee
Rel_ID=12, target_cpu=0
```
3. Unbind the device, using the `Device_ID` from the previous command's output:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ echo | sudo tee /sys/bus/vmbus/drivers/hv_utils/unbind
```
4. Install the `ntp` package:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo apt-get install ntp
```
5. Stop the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp stop
```
6. Sync the machine's clock with Google's NTP service:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpd -b time.google.com
```
To make this change permanent, in the `/etc/ntp.conf` file, remove or comment out any lines starting with `server` or `pool` and add the following lines:
```
server time1.google.com iburst
server time2.google.com iburst
server time3.google.com iburst
server time4.google.com iburst
```
Restart the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp start
```
We recommend Google's NTP service because it handles ["smearing" the leap second](https://developers.google.com/time/smear). If you use a different NTP service that doesn't smear the leap second, be sure to configure client-side smearing in the same way on each machine. See the Production Checklist for details.
7. Verify that the machine is using a Google NTP server:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpq -p
```
The active NTP server will be marked with an asterisk.
8. Repeat these steps for each machine where a CockroachDB node will run.
## Step 4. Set up load balancing
Each CockroachDB node is an equally suitable SQL gateway to your cluster, but to ensure client performance and reliability, it's important to use load balancing:
* **Performance:** Load balancers spread client traffic across nodes. This prevents any one node from being overwhelmed by requests and improves overall cluster performance (queries per second).
* **Reliability:** Load balancers decouple client health from the health of a single CockroachDB node. In cases where a node fails, the load balancer redirects client traffic to available nodes.
AWS offers fully-managed load balancing to distribute traffic between instances.
1. [Add AWS load balancing](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/network-load-balancer-getting-started.html). Be sure to:
* Select a **Network Load Balancer** and use the ports we specify below.
* Select the VPC and *all* availability zones of your instances. This is important, as you cannot change the availability zones once the load balancer is created. The availability zone of an instance is determined by its subnet, found by inspecting the instance in the Amazon EC2 Console.
* Set the load balancer port to **26257**.
* Create a new target group that uses TCP port **26257**. Traffic from your load balancer is routed to this target group, which contains your instances.
* Configure health checks to use HTTP port **8080** and path `/health?ready=1`. This health endpoint ensures that load balancers do not direct traffic to nodes that are live but not ready to receive requests.
* Register your instances with the target group you created, specifying port **26257**. You can add and remove instances later.
2. To test load balancing and connect your application to the cluster, you will need the provisioned internal (private) **IP address** for the load balancer. To find this, open the Network Interfaces section of the Amazon EC2 console and look up the load balancer by its name.
## Step 5. Generate certificates
You can use `cockroach cert` commands or `openssl` commands to generate security certificates. This section features the `cockroach cert` commands.
Locally, you'll need to create the following certificates and keys:
* A certificate authority (CA) key pair (`ca.crt` and `ca.key`).
* A node key pair for each node, issued to its IP addresses and any common names the machine uses, as well as to the IP addresses and common names for machines running load balancers.
* A client key pair for the `root` user. You'll use this to run a sample workload against the cluster as well as some `cockroach` client commands from your local machine.
1. Install CockroachDB on your local machine, if you haven't already.
2. Create two directories:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir certs
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir my-safe-directory
```
* `certs`: You'll generate your CA certificate and all node and client certificates and keys in this directory and then upload some of the files to your nodes.
* `my-safe-directory`: You'll generate your CA key in this directory and then reference the key when generating node and client certificates. After that, you'll keep the key safe and secret; you will not upload it to your nodes.
3. Create the CA certificate and key:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-ca \
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
4. Create the certificate and key for the first node, issued to all common names you might use to refer to the node as well as to the load balancer instances:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-node \
\
\
\
\
localhost \
127.0.0.1 \
\
\
\
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
5. Upload the CA certificate and node certificate and key to the first node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ gcloud compute ssh \
--project \
--command "mkdir certs"
```
`gcloud compute ssh` associates your public SSH key with the GCP project and is only needed when connecting to the first node. See the [GCP docs](https://cloud.google.com/sdk/gcloud/reference/compute/ssh) for more details.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ scp certs/ca.crt \
certs/node.crt \
certs/node.key \
@@@@@
This is necessary because the certificates and keys for additional nodes will also be named `node.crt` and `node.key`. As an alternative to deleting these files, you can run the next `cockroach cert create-node` commands with the `--overwrite` flag.
1. Create the certificate and key for the second node, issued to all common names you might use to refer to the node as well as to the load balancer instances:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-node \
\
\
\
\
localhost \
127.0.0.1 \
\
\
\
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
2. Upload the CA certificate and node certificate and key to the second node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ ssh @@@@@@`cockroach` client commands from your local machine. If you might also want to run `cockroach` client commands directly on a node (e.g., for local debugging), you'll need to copy the `root` user's certificate and key to that node as well.
On accessing the DB Console in a later step, your browser will consider the CockroachDB-created certificate invalid and you’ll need to click through a warning message to get to the UI. You can avoid this issue by using a certificate issued by a public CA.
## Step 6. Start nodes
You can start the nodes manually or automate the process using [systemd](https://www.freedesktop.org/wiki/Software/systemd/).
For each initial node of your cluster, complete the following steps:
After completing these steps, nodes will not yet be live. They will complete the startup process and join together to form a cluster as soon as the cluster is initialized in the next step.
1. Visit Releases and download the full binary of CockroachDB to the node.
2. On the node, follow the instructions to install CockroachDB.
3. Run the `cockroach start` command:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--advertise-addr=
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
| `--cache`
`--max-sql-memory` | Increases the node's cache size to 25% of available system memory to improve read performance. The capacity for in-memory SQL processing defaults to 25% of system memory but can be raised, if necessary, to increase the number of simultaneous client connections allowed by the node as well as the node's capacity for in-memory processing of rows when using `ORDER BY`, `GROUP BY`, `DISTINCT`, joins, and window functions. For more details, see Cache and SQL Memory Size. |
| `--background` | Starts the node in the background so you gain control of the terminal to issue more commands. |
When deploying across multiple datacenters, or when there is otherwise high latency between nodes, it is recommended to set `--locality` as well. For more details, see Locality.
For other flags not explicitly set, the command uses default values. For example, the node stores data in `--store=cockroach-data` and binds DB Console HTTP requests to `--http-addr=Start a Node.
Repeat these steps for each additional node that you want in your cluster.
For each initial node of your cluster, complete the following steps:
After completing these steps, nodes will not yet be live. They will complete the startup process and join together to form a cluster as soon as the cluster is initialized in the next step.
1. SSH to the machine where you want the node to run. Ensure you are logged in as the `root` user.
2. Install CockroachDB for Linux.
3. Create the Cockroach directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir /var/lib/cockroach
```
4. Create a Unix user named `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ useradd cockroach
```
5. Move the `certs` directory to the `cockroach` directory.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mv certs /var/lib/cockroach/
```
6. Change the ownership of the `cockroach` directory to the user `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ chown -R cockroach /var/lib/cockroach
```
7. Download the [sample configuration template](https://raw.githubusercontent.com/cockroachlabs/docs/main/src/current/_includes/v26.3/prod-deployment/securecockroachdb.service) and save the file in the `/etc/systemd/system/` directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl -o securecockroachdb.service https://raw.githubusercontent.com/cockroachlabs/docs/main/src/current/_includes/v23.2/prod-deployment/securecockroachdb.service
```
Alternatively, you can create the file yourself and copy the script into it:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
[Unit]
Description=Cockroach Database cluster node
Requires=network.target
[Service]
Type=notify
WorkingDirectory=/var/lib/cockroach
ExecStart=/usr/local/bin/cockroach start --certs-dir=certs --advertise-addr=\
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
When deploying across multiple datacenters, or when there is otherwise high latency between nodes, it is recommended to set `--locality` as well. For more details, see Locality.
For other flags not explicitly set, the command uses default values. For example, the node stores data in `--store=cockroach-data` and binds DB Console HTTP requests to `--http-addr=localhost:8080`. To set these options manually, see Start a Node.
9. Start the CockroachDB cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
systemctl start securecockroachdb
```
10. Configure `systemd` to start CockroachDB automatically after a reboot:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
systemctl enable securecockroachdb
```
11. Repeat these steps for each additional node that you want in your cluster.
`systemd` handles node restarts in case of node failure. To stop a node without `systemd` restarting it, run `systemctl stop securecockroachdb`
## Step 7. Initialize the cluster
On your local machine, run the `cockroach init` command to complete the node startup process and have them join together as a cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach init --certs-dir=certs --host=standard output, such as the CockroachDB version, the URL for the DB Console, and the SQL URL for clients.
## Step 8. Test your cluster
CockroachDB replicates and distributes data behind-the-scenes and uses a [Gossip protocol](https://wikipedia.org/wiki/Gossip_protocol) to enable each node to locate data across the cluster. Once a cluster is live, any node can be used as a SQL gateway.
When using a load balancer, you should issue commands directly to the load balancer, which then routes traffic to the nodes.
Use the built-in SQL client locally as follows:
1. On your local machine, launch the built-in SQL client, with the `--host` flag set to the address of the load balancer:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --certs-dir=certs --host= CREATE DATABASE securenodetest;
```
3. View the cluster's databases, which will include `securenodetest`:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SHOW DATABASES;
```
```
+--------------------+
| Database |
+--------------------+
| crdb_internal |
| information_schema |
| securenodetest |
| pg_catalog |
| system |
+--------------------+
(5 rows)
```
4. Use `\q` to exit the SQL shell.
## Step 9. Run a sample workload
CockroachDB comes with a number of built-in workloads for simulating client traffic. This step features CockroachDB's version of the [TPC-C](http://www.tpc.org/tpcc/) workload.
Be sure that you have configured your network to allow traffic from the application to the load balancer. In this case, you will run the sample workload on one of your machines. The traffic source should therefore be the **internal (private)** IP address of that machine.
1. SSH to the machine where you want to run the sample TPC-C workload.
This should be a machine that is not running a CockroachDB node, and it should already have a `certs` directory containing `ca.crt`, `client.root.crt`, and `client.root.key` files.
2. Install CockroachDB for Linux.
3. Use the `cockroach workload` command to load the initial schema and data, pointing it at the IP address of the load balancer:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload init tpcc \
'postgresql://root@:26257/tpcc?sslmode=verify-full&sslrootcert=certs/ca.crt&sslcert=certs/client.root.crt&sslkey=certs/client.root.key'
```
4. Use the `cockroach workload` command to run the workload for 10 minutes:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload run tpcc \
--duration=10m \
'postgresql://root@:26257/tpcc?sslmode=verify-full&sslrootcert=certs/ca.crt&sslcert=certs/client.root.crt&sslkey=certs/client.root.key'
```
You'll see per-operation statistics print to standard output every second:
```
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
1s 0 1443.4 1494.8 4.7 9.4 27.3 67.1 transfer
2s 0 1686.5 1590.9 4.7 8.1 15.2 28.3 transfer
3s 0 1735.7 1639.0 4.7 7.3 11.5 28.3 transfer
4s 0 1542.6 1614.9 5.0 8.9 12.1 21.0 transfer
5s 0 1695.9 1631.1 4.7 7.3 11.5 22.0 transfer
6s 0 1569.2 1620.8 5.0 8.4 11.5 15.7 transfer
7s 0 1614.6 1619.9 4.7 8.1 12.1 16.8 transfer
8s 0 1344.4 1585.6 5.8 10.0 15.2 31.5 transfer
9s 0 1351.9 1559.5 5.8 10.0 16.8 54.5 transfer
10s 0 1514.8 1555.0 5.2 8.1 12.1 16.8 transfer
...
```
After the specified duration (10 minutes in this case), the workload will stop and you'll see totals printed to standard output:
```
_elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__result
600.0s 0 823902 1373.2 5.8 5.5 10.0 15.2 209.7
```
For more `tpcc` options, use `cockroach workload run tpcc --help`. For details about other workloads built into the `cockroach` binary, use `cockroach workload --help`.
1. To monitor the load generator's progress, open the DB Console by pointing a browser to the address in the `admin` field in the standard output of any node on startup.
Since the load generator is pointed at the load balancer, the connections will be evenly distributed across nodes. To verify this, click **Metrics** on the left, select the **SQL** dashboard, and then check the **SQL Connections** graph. You can use the **Graph** menu to filter the graph for specific nodes.
## Step 10. Monitor the cluster
In the Target Groups section of the Amazon EC2 console, [check the health](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-health-checks.html) of your instances by inspecting your target group and opening the Targets tab.
Despite CockroachDB's various built-in safeguards against failure, it is critical to actively monitor the overall health and performance of a cluster running in production and to create alerting rules that promptly send notifications when there are events that require investigation or intervention.
For details about available monitoring options and the most important events and metrics to alert on, see Monitoring and Alerting.
## Step 11. Scale the cluster
Before adding a new node, [create a new instance](#step-1-create-instances) as you did earlier. Then [generate and upload a certificate and key](#step-5-generate-certificates) for the new node.
You can start the nodes manually or automate the process using [systemd](https://www.freedesktop.org/wiki/Software/systemd/).
For each additional node you want to add to the cluster, complete the following steps:
1. SSH to the machine where you want the node to run.
2. Install CockroachDB for Linux.
3. Run the `cockroach start` command, passing the new node's address as the `--advertise-addr` flag and pointing `--join` to the three existing nodes (also include `--locality` if you set it earlier).
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--advertise-addr=
For each additional node you want to add to the cluster, complete the following steps:
1. SSH to the machine where you want the node to run. Ensure you are logged in as the `root` user.
2. Install CockroachDB for Linux.
3. Create the Cockroach directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
mkdir /var/lib/cockroach
```
4. Create a Unix user named `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
useradd cockroach
```
5. Move the `certs` directory to the `cockroach` directory.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
mv certs /var/lib/cockroach/
```
6. Change the ownership of the `cockroach` directory to the user `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
chown -R cockroach /var/lib/cockroach
```
7. Download the [sample configuration template](https://raw.githubusercontent.com/cockroachlabs/docs/master/_includes/v26.3/prod-deployment/securecockroachdb.service):
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl -o securecockroachdb.service https://raw.githubusercontent.com/cockroachlabs/docs/master/_includes/v26.3/prod-deployment/securecockroachdb.service
```
Alternatively, you can create the file yourself and copy the script into it:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
[Unit]
Description=Cockroach Database cluster node
Requires=network.target
[Service]
Type=notify
WorkingDirectory=/var/lib/cockroach
ExecStart=/usr/local/bin/cockroach start --certs-dir=certs --advertise-addr=\
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
9. Repeat these steps for each additional node that you want in your cluster.
## Step 12. Use the database
Now that your deployment is working, you can:
1. Implement your data model.
2. Create users and grant them privileges.
3. Connect your application. Be sure to connect your application to the load balancer, not to a CockroachDB node.
4. Take backups of your data.
You may also want to adjust the way the cluster replicates data. For example, by default, a multi-node cluster replicates all data 3 times; you can change this replication factor or create additional rules for replicating individual databases and tables differently. For more information, see Replication Controls.
When running a cluster of 5 nodes or more, it's safest to increase the replication factor for important internal data to 5, even if you do not do so for user data. For the cluster as a whole to remain available, the ranges for this internal data must always retain a majority of their replicas.
## See also
* Production Checklist
* Manual Deployment
* Orchestrated Deployment
* Monitoring and Alerting
* Performance Tuning
* Local Deployment
# Deploy CockroachDB on Digital Ocean
Source: https://docs.cockroachlabs.com/docs/v26.3/deploy-cockroachdb-on-digital-ocean
This page shows you how to deploy a secure multi-node CockroachDB cluster on Digital Ocean, using Digital Ocean's managed load balancing service to distribute client traffic.
If you are only testing CockroachDB, or you are not concerned with protecting network communication with TLS encryption, you can use an insecure cluster instead. Select **Insecure** above for instructions.
To try CockroachDB Cloud instead of running CockroachDB yourself, refer to the Cloud Quickstart.
## Before you begin
### Requirements
* You must have CockroachDB installed locally. This is necessary for generating and managing your deployment's certificates.
* You must have [SSH access]() to each machine. This is necessary for distributing and starting CockroachDB binaries.
* Your network configuration must allow TCP communication on the following ports:
* `26257` for intra-cluster and client-cluster communication
* `8080` to expose your DB Console
* Carefully review the Production Checklist, including supported hardware and software, and the recommended Topology Patterns.
* Do not run multiple node processes on the same VM or machine. This defeats CockroachDB's replication and causes the system to be a single point of failure. Instead, start each node on a separate VM or machine.
* To start a node with multiple disks or SSDs, provide a separate `--store` flag for each disk when starting the `cockroach` process on the node. For more details about stores, see Start a Node.
If you start a node with multiple `--store` flags, it is not possible to scale back down to only using a single store on the node. Instead, you must decommission the node and start a new node with the updated `--store`.
* When starting each node, use the `--locality` flag to describe the node's location, for example, `--locality=region=west,zone=us-west-1`. The key-value pairs should be ordered from most to least inclusive, and the keys and order of key-value pairs must be the same on all nodes.
* When deploying in a single availability zone:
* To be able to tolerate the failure of any 1 node, use at least 3 nodes with the `default` 3-way replication factor. In this case, if 1 node fails, each range retains 2 of its 3 replicas, a majority.
* To be able to tolerate 2 simultaneous node failures, use at least 5 nodes and increase the `default` replication factor for user data to 5. The replication factor for important internal data is 5 by default, so no adjustments are needed for internal data. In this case, if 2 nodes fail at the same time, each range retains 3 of its 5 replicas, a majority.
* When deploying across multiple availability zones:
* To be able to tolerate the failure of 1 entire AZ in a region, use at least 3 AZs per region and set `--locality` on each node to spread data evenly across regions and AZs. In this case, if 1 AZ goes offline, the 2 remaining AZs retain a majority of replicas.
* To ensure that ranges are split evenly across nodes, use the same number of nodes in each AZ. This is to avoid overloading any nodes with excessive resource consumption.
* When deploying across multiple regions:
* To be able to tolerate the failure of 1 entire region, use at least 3 regions.
### Recommendations
* Decide how you want to access your DB Console:
| Access Level | Description |
| - | - |
| Partially open | Set a firewall rule to allow only specific IP addresses to communicate on port `8080`. |
| Completely open | Set a firewall rule to allow all IP addresses to communicate on port `8080`. |
| Completely closed | Set a firewall rule to disallow all communication on port `8080`. In this case, a machine with SSH access to a node could use an SSH tunnel to access the DB Console. |
* If all of your CockroachDB nodes and clients will run on Droplets in a single region, consider using [private networking](https://docs.digitalocean.com/products/networking/vpc/how-to/create/).
## Step 1. Create Droplets
[Create Droplets](https://www.digitalocean.com/community/tutorials/how-to-create-your-first-digitalocean-droplet) for each node you plan to have in your cluster. If you plan to run a sample workload against the cluster, create a separate Droplet for that workload.
* Run at least 3 nodes to ensure survivability.
* Use any [droplets](https://www.digitalocean.com/pricing/) except standard droplets with only 1 GB of RAM, which is below our minimum requirement. All Digital Ocean droplets use SSD storage.
For more details, see Hardware Recommendations and Cluster Topology.
## Step 2. Synchronize clocks
CockroachDB requires moderate levels of clock synchronization to preserve data consistency. For this reason, when a node detects that its clock is out of sync with at least half of the other nodes in the cluster by 80% of the maximum offset allowed (500ms by default), it spontaneously shuts down. This avoids the risk of consistency anomalies, but it's best to prevent clocks from drifting too far in the first place by running clock synchronization software on each node.
[`ntpd`](http://doc.ntp.org/) should keep offsets in the single-digit milliseconds, so that software is featured here, but other methods of clock synchronization are suitable as well.
1. SSH to the first machine.
2. Disable `timesyncd`, which tends to be active by default on some Linux distributions:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo timedatectl set-ntp no
```
Verify that `timesyncd` is off:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ timedatectl
```
Look for `Network time on: no` or `NTP enabled: no` in the output.
3. Install the `ntp` package:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo apt-get install ntp
```
4. Stop the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp stop
```
5. Sync the machine's clock with Google's NTP service:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpd -b time.google.com
```
To make this change permanent, in the `/etc/ntp.conf` file, remove or comment out any lines starting with `server` or `pool` and add the following lines:
```
server time1.google.com iburst
server time2.google.com iburst
server time3.google.com iburst
server time4.google.com iburst
```
Restart the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp start
```
We recommend Google's NTP service because it handles ["smearing" the leap second](https://developers.google.com/time/smear). If you use a different NTP service that doesn't smear the leap second, be sure to configure client-side smearing in the same way on each machine. See the Production Checklist for details.
1. Verify that the machine is using a Google NTP server:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpq -p
```
The active NTP server will be marked with an asterisk.
2. Repeat these steps for each machine where a CockroachDB node will run.
Compute Engine instances are preconfigured to use [NTP](http://www.ntp.org/), which should keep offsets in the single-digit milliseconds. However, Google can’t predict how external NTP services, such as `pool.ntp.org`, will handle the leap second. Therefore, you should:
* [Configure each GCE instance to use Google's internal NTP service](https://cloud.google.com/compute/docs/instances/configure-ntp#configure_ntp_for_your_instances).
* If you plan to run a hybrid cluster across GCE and other cloud providers or environments, note that all of the nodes must be synced to the same time source, or to different sources that implement leap second smearing in the same way. See the Production Checklist for details.
Amazon provides the [Amazon Time Sync Service](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html), which uses a fleet of satellite-connected and atomic reference clocks in each AWS Region to deliver accurate current time readings. The service also smears the leap second.
* [Configure each AWS instance to use the internal Amazon Time Sync Service](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html#configure-amazon-time-service).
* Per the above instructions, ensure that `etc/chrony.conf` on the instance contains the line `server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4` and that other `server` or `pool` lines are commented out.
* To verify that Amazon Time Sync Service is being used, run `chronyc sources -v` and check for a line containing `* 169.254.169.123`. The `*` denotes the preferred time server.
* If you plan to run a hybrid cluster across GCE and other cloud providers or environments, note that all of the nodes must be synced to the same time source, or to different sources that implement leap second smearing in the same way. See the Production Checklist for details.
[`ntpd`](http://doc.ntp.org/) should keep offsets in the single-digit milliseconds, so that software is featured here. However, to run `ntpd` properly on Azure VMs, it's necessary to first unbind the Time Synchronization device used by the Hyper-V technology running Azure VMs; this device aims to synchronize time between the VM and its host operating system but has been known to cause problems.
1. SSH to the first machine.
2. Find the ID of the Hyper-V Time Synchronization device:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ curl -O https://raw.githubusercontent.com/torvalds/linux/master/tools/hv/lsvmbus
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ python lsvmbus -vv | grep -w "Time Synchronization" -A 3
```
```
VMBUS ID 12: Class_ID = {9527e630-d0ae-497b-adce-e80ab0175caf} - [Time Synchronization]
Device_ID = {2dd1ce17-079e-403c-b352-a1921ee207ee}
Sysfs path: /sys/bus/vmbus/devices/2dd1ce17-079e-403c-b352-a1921ee207ee
Rel_ID=12, target_cpu=0
```
3. Unbind the device, using the `Device_ID` from the previous command's output:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ echo | sudo tee /sys/bus/vmbus/drivers/hv_utils/unbind
```
4. Install the `ntp` package:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo apt-get install ntp
```
5. Stop the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp stop
```
6. Sync the machine's clock with Google's NTP service:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpd -b time.google.com
```
To make this change permanent, in the `/etc/ntp.conf` file, remove or comment out any lines starting with `server` or `pool` and add the following lines:
```
server time1.google.com iburst
server time2.google.com iburst
server time3.google.com iburst
server time4.google.com iburst
```
Restart the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp start
```
We recommend Google's NTP service because it handles ["smearing" the leap second](https://developers.google.com/time/smear). If you use a different NTP service that doesn't smear the leap second, be sure to configure client-side smearing in the same way on each machine. See the Production Checklist for details.
7. Verify that the machine is using a Google NTP server:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpq -p
```
The active NTP server will be marked with an asterisk.
8. Repeat these steps for each machine where a CockroachDB node will run.
## Step 3. Set up load balancing
Each CockroachDB node is an equally suitable SQL gateway to your cluster, but to ensure client performance and reliability, it's important to use load balancing:
* **Performance:** Load balancers spread client traffic across nodes. This prevents any one node from being overwhelmed by requests and improves overall cluster performance (queries per second).
* **Reliability:** Load balancers decouple client health from the health of a single CockroachDB node. In cases where a node fails, the load balancer redirects client traffic to available nodes.
Digital Ocean offers fully-managed load balancers to distribute traffic between Droplets.
1. [Create a Digital Ocean Load Balancer](https://www.digitalocean.com/community/tutorials/an-introduction-to-digitalocean-load-balancers). Be sure to:
* Set forwarding rules to route TCP traffic from the load balancer's port **26257** to port **26257** on the node Droplets.
* Configure health checks to use HTTP port **8080** and path `/health?ready=1`. This health endpoint ensures that load balancers do not direct traffic to nodes that are live but not ready to receive requests.
2. Note the provisioned **IP Address** for the load balancer. You'll use this later to test load balancing and to connect your application to the cluster.
## Step 4. Configure your network
Set up a firewall for each of your Droplets, allowing TCP communication on the following two ports:
* **26257** (`tcp:26257`) for inter-node communication (i.e., working as a cluster), for applications to connect to the load balancer, and for routing from the load balancer to nodes
* **8080** (`tcp:8080`) for exposing your DB Console
For guidance, you can use Digital Ocean's guide to configuring firewalls based on the Droplet's OS:
* Ubuntu and Debian can use [`ufw`](https://www.digitalocean.com/community/tutorials/how-to-setup-a-firewall-with-ufw-on-an-ubuntu-and-debian-cloud-server).
* FreeBSD can use [`ipfw`](https://www.digitalocean.com/community/tutorials/recommended-steps-for-new-freebsd-10-1-servers).
* Fedora can use [`iptables`](https://www.digitalocean.com/community/tutorials/initial-setup-of-a-fedora-22-server).
* CoreOS can use [`iptables`](https://www.digitalocean.com/community/tutorials/how-to-secure-your-coreos-cluster-with-tls-ssl-and-firewall-rules).
* CentOS can use [`firewalld`](https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-using-firewalld-on-centos-7).
## Step 5. Generate certificates
You can use `cockroach cert` commands or `openssl` commands to generate security certificates. This section features the `cockroach cert` commands.
Locally, you'll need to create the following certificates and keys:
* A certificate authority (CA) key pair (`ca.crt` and `ca.key`).
* A node key pair for each node, issued to its IP addresses and any common names the machine uses, as well as to the IP addresses and common names for machines running load balancers.
* A client key pair for the `root` user. You'll use this to run a sample workload against the cluster as well as some `cockroach` client commands from your local machine.
1. Install CockroachDB on your local machine, if you haven't already.
2. Create two directories:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir certs
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir my-safe-directory
```
* `certs`: You'll generate your CA certificate and all node and client certificates and keys in this directory and then upload some of the files to your nodes.
* `my-safe-directory`: You'll generate your CA key in this directory and then reference the key when generating node and client certificates. After that, you'll keep the key safe and secret; you will not upload it to your nodes.
3. Create the CA certificate and key:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-ca \
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
4. Create the certificate and key for the first node, issued to all common names you might use to refer to the node as well as to the load balancer instances:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-node \
\
\
\
\
localhost \
127.0.0.1 \
\
\
\
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
5. Upload the CA certificate and node certificate and key to the first node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ gcloud compute ssh \
--project \
--command "mkdir certs"
```
`gcloud compute ssh` associates your public SSH key with the GCP project and is only needed when connecting to the first node. See the [GCP docs](https://cloud.google.com/sdk/gcloud/reference/compute/ssh) for more details.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ scp certs/ca.crt \
certs/node.crt \
certs/node.key \
@@@@@
This is necessary because the certificates and keys for additional nodes will also be named `node.crt` and `node.key`. As an alternative to deleting these files, you can run the next `cockroach cert create-node` commands with the `--overwrite` flag.
1. Create the certificate and key for the second node, issued to all common names you might use to refer to the node as well as to the load balancer instances:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-node \
\
\
\
\
localhost \
127.0.0.1 \
\
\
\
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
2. Upload the CA certificate and node certificate and key to the second node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ ssh @@@@@@`cockroach` client commands from your local machine. If you might also want to run `cockroach` client commands directly on a node (e.g., for local debugging), you'll need to copy the `root` user's certificate and key to that node as well.
On accessing the DB Console in a later step, your browser will consider the CockroachDB-created certificate invalid and you’ll need to click through a warning message to get to the UI. You can avoid this issue by using a certificate issued by a public CA.
## Step 6. Start nodes
You can start the nodes manually or automate the process using [systemd](https://www.freedesktop.org/wiki/Software/systemd/).
For each initial node of your cluster, complete the following steps:
After completing these steps, nodes will not yet be live. They will complete the startup process and join together to form a cluster as soon as the cluster is initialized in the next step.
1. Visit Releases and download the full binary of CockroachDB to the node.
2. On the node, follow the instructions to install CockroachDB.
3. Run the `cockroach start` command:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--advertise-addr=
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
| `--cache`
`--max-sql-memory` | Increases the node's cache size to 25% of available system memory to improve read performance. The capacity for in-memory SQL processing defaults to 25% of system memory but can be raised, if necessary, to increase the number of simultaneous client connections allowed by the node as well as the node's capacity for in-memory processing of rows when using `ORDER BY`, `GROUP BY`, `DISTINCT`, joins, and window functions. For more details, see Cache and SQL Memory Size. |
| `--background` | Starts the node in the background so you gain control of the terminal to issue more commands. |
When deploying across multiple datacenters, or when there is otherwise high latency between nodes, it is recommended to set `--locality` as well. For more details, see Locality.
For other flags not explicitly set, the command uses default values. For example, the node stores data in `--store=cockroach-data` and binds DB Console HTTP requests to `--http-addr=Start a Node.
Repeat these steps for each additional node that you want in your cluster.
For each initial node of your cluster, complete the following steps:
After completing these steps, nodes will not yet be live. They will complete the startup process and join together to form a cluster as soon as the cluster is initialized in the next step.
1. SSH to the machine where you want the node to run. Ensure you are logged in as the `root` user.
2. Install CockroachDB for Linux.
3. Create the Cockroach directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir /var/lib/cockroach
```
4. Create a Unix user named `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ useradd cockroach
```
5. Move the `certs` directory to the `cockroach` directory.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mv certs /var/lib/cockroach/
```
6. Change the ownership of the `cockroach` directory to the user `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ chown -R cockroach /var/lib/cockroach
```
7. Download the [sample configuration template](https://raw.githubusercontent.com/cockroachlabs/docs/main/src/current/_includes/v26.3/prod-deployment/securecockroachdb.service) and save the file in the `/etc/systemd/system/` directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl -o securecockroachdb.service https://raw.githubusercontent.com/cockroachlabs/docs/main/src/current/_includes/v23.2/prod-deployment/securecockroachdb.service
```
Alternatively, you can create the file yourself and copy the script into it:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
[Unit]
Description=Cockroach Database cluster node
Requires=network.target
[Service]
Type=notify
WorkingDirectory=/var/lib/cockroach
ExecStart=/usr/local/bin/cockroach start --certs-dir=certs --advertise-addr=\
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
When deploying across multiple datacenters, or when there is otherwise high latency between nodes, it is recommended to set `--locality` as well. For more details, see Locality.
For other flags not explicitly set, the command uses default values. For example, the node stores data in `--store=cockroach-data` and binds DB Console HTTP requests to `--http-addr=localhost:8080`. To set these options manually, see Start a Node.
9. Start the CockroachDB cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
systemctl start securecockroachdb
```
10. Configure `systemd` to start CockroachDB automatically after a reboot:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
systemctl enable securecockroachdb
```
11. Repeat these steps for each additional node that you want in your cluster.
`systemd` handles node restarts in case of node failure. To stop a node without `systemd` restarting it, run `systemctl stop securecockroachdb`
## Step 7. Initialize the cluster
On your local machine, run the `cockroach init` command to complete the node startup process and have them join together as a cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach init --certs-dir=certs --host=standard output, such as the CockroachDB version, the URL for the DB Console, and the SQL URL for clients.
## Step 8. Test the cluster
CockroachDB replicates and distributes data behind-the-scenes and uses a [Gossip protocol](https://wikipedia.org/wiki/Gossip_protocol) to enable each node to locate data across the cluster. Once a cluster is live, any node can be used as a SQL gateway.
When using a load balancer, you should issue commands directly to the load balancer, which then routes traffic to the nodes.
Use the built-in SQL client locally as follows:
1. On your local machine, launch the built-in SQL client, with the `--host` flag set to the address of the load balancer:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --certs-dir=certs --host= CREATE DATABASE securenodetest;
```
3. View the cluster's databases, which will include `securenodetest`:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SHOW DATABASES;
```
```
+--------------------+
| Database |
+--------------------+
| crdb_internal |
| information_schema |
| securenodetest |
| pg_catalog |
| system |
+--------------------+
(5 rows)
```
4. Use `\q` to exit the SQL shell.
## Step 9. Run a sample workload
CockroachDB comes with a number of built-in workloads for simulating client traffic. This step features CockroachDB's version of the [TPC-C](http://www.tpc.org/tpcc/) workload.
Be sure that you have configured your network to allow traffic from the application to the load balancer. In this case, you will run the sample workload on one of your machines. The traffic source should therefore be the **internal (private)** IP address of that machine.
1. SSH to the machine where you want to run the sample TPC-C workload.
This should be a machine that is not running a CockroachDB node, and it should already have a `certs` directory containing `ca.crt`, `client.root.crt`, and `client.root.key` files.
2. Install CockroachDB for Linux.
3. Use the `cockroach workload` command to load the initial schema and data, pointing it at the IP address of the load balancer:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload init tpcc \
'postgresql://root@:26257/tpcc?sslmode=verify-full&sslrootcert=certs/ca.crt&sslcert=certs/client.root.crt&sslkey=certs/client.root.key'
```
4. Use the `cockroach workload` command to run the workload for 10 minutes:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload run tpcc \
--duration=10m \
'postgresql://root@:26257/tpcc?sslmode=verify-full&sslrootcert=certs/ca.crt&sslcert=certs/client.root.crt&sslkey=certs/client.root.key'
```
You'll see per-operation statistics print to standard output every second:
```
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
1s 0 1443.4 1494.8 4.7 9.4 27.3 67.1 transfer
2s 0 1686.5 1590.9 4.7 8.1 15.2 28.3 transfer
3s 0 1735.7 1639.0 4.7 7.3 11.5 28.3 transfer
4s 0 1542.6 1614.9 5.0 8.9 12.1 21.0 transfer
5s 0 1695.9 1631.1 4.7 7.3 11.5 22.0 transfer
6s 0 1569.2 1620.8 5.0 8.4 11.5 15.7 transfer
7s 0 1614.6 1619.9 4.7 8.1 12.1 16.8 transfer
8s 0 1344.4 1585.6 5.8 10.0 15.2 31.5 transfer
9s 0 1351.9 1559.5 5.8 10.0 16.8 54.5 transfer
10s 0 1514.8 1555.0 5.2 8.1 12.1 16.8 transfer
...
```
After the specified duration (10 minutes in this case), the workload will stop and you'll see totals printed to standard output:
```
_elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__result
600.0s 0 823902 1373.2 5.8 5.5 10.0 15.2 209.7
```
For more `tpcc` options, use `cockroach workload run tpcc --help`. For details about other workloads built into the `cockroach` binary, use `cockroach workload --help`.
1. To monitor the load generator's progress, open the DB Console by pointing a browser to the address in the `admin` field in the standard output of any node on startup.
Since the load generator is pointed at the load balancer, the connections will be evenly distributed across nodes. To verify this, click **Metrics** on the left, select the **SQL** dashboard, and then check the **SQL Connections** graph. You can use the **Graph** menu to filter the graph for specific nodes.
## Step 10. Monitor the cluster
Despite CockroachDB's various built-in safeguards against failure, it is critical to actively monitor the overall health and performance of a cluster running in production and to create alerting rules that promptly send notifications when there are events that require investigation or intervention.
For details about available monitoring options and the most important events and metrics to alert on, see Monitoring and Alerting.
## Step 11. Scale the cluster
You can start the nodes manually or automate the process using [systemd](https://www.freedesktop.org/wiki/Software/systemd/).
For each additional node you want to add to the cluster, complete the following steps:
1. SSH to the machine where you want the node to run.
2. Install CockroachDB for Linux.
3. Run the `cockroach start` command, passing the new node's address as the `--advertise-addr` flag and pointing `--join` to the three existing nodes (also include `--locality` if you set it earlier).
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--advertise-addr=
For each additional node you want to add to the cluster, complete the following steps:
1. SSH to the machine where you want the node to run. Ensure you are logged in as the `root` user.
2. Install CockroachDB for Linux.
3. Create the Cockroach directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
mkdir /var/lib/cockroach
```
4. Create a Unix user named `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
useradd cockroach
```
5. Move the `certs` directory to the `cockroach` directory.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
mv certs /var/lib/cockroach/
```
6. Change the ownership of the `cockroach` directory to the user `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
chown -R cockroach /var/lib/cockroach
```
7. Download the [sample configuration template](https://raw.githubusercontent.com/cockroachlabs/docs/master/_includes/v26.3/prod-deployment/securecockroachdb.service):
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl -o securecockroachdb.service https://raw.githubusercontent.com/cockroachlabs/docs/master/_includes/v26.3/prod-deployment/securecockroachdb.service
```
Alternatively, you can create the file yourself and copy the script into it:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
[Unit]
Description=Cockroach Database cluster node
Requires=network.target
[Service]
Type=notify
WorkingDirectory=/var/lib/cockroach
ExecStart=/usr/local/bin/cockroach start --certs-dir=certs --advertise-addr=\
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
9. Repeat these steps for each additional node that you want in your cluster.
## Step 12. Use the database
Now that your deployment is working, you can:
1. Implement your data model.
2. Create users and grant them privileges.
3. Connect your application. Be sure to connect your application to the load balancer, not to a CockroachDB node.
4. Take backups of your data.
You may also want to adjust the way the cluster replicates data. For example, by default, a multi-node cluster replicates all data 3 times; you can change this replication factor or create additional rules for replicating individual databases and tables differently. For more information, see Replication Controls.
When running a cluster of 5 nodes or more, it's safest to increase the replication factor for important internal data to 5, even if you do not do so for user data. For the cluster as a whole to remain available, the ranges for this internal data must always retain a majority of their replicas.
## See also
* Production Checklist
* Manual Deployment
* Orchestrated Deployment
* Monitoring and Alerting
* Performance Tuning
* Local Deployment
# Deploy CockroachDB on Google Cloud Platform GCE
Source: https://docs.cockroachlabs.com/docs/v26.3/deploy-cockroachdb-on-google-cloud-platform
This page shows you how to manually deploy a secure multi-node CockroachDB cluster on Google Cloud Platform's Compute Engine (GCE), using Google's TCP Proxy Load Balancing service to distribute client traffic.
If you are only testing CockroachDB, or you are not concerned with protecting network communication with TLS encryption, you can use an insecure cluster instead. Select **Insecure** above for instructions.
To try CockroachDB Cloud instead of running CockroachDB yourself, refer to the Cloud Quickstart.
## Before you begin
### Requirements
* You must have CockroachDB installed locally. This is necessary for generating and managing your deployment's certificates.
* You must have [SSH access]() to each machine. This is necessary for distributing and starting CockroachDB binaries.
* Your network configuration must allow TCP communication on the following ports:
* `26257` for intra-cluster and client-cluster communication
* `8080` to expose your DB Console
* Carefully review the Production Checklist, including supported hardware and software, and the recommended Topology Patterns.
* Do not run multiple node processes on the same VM or machine. This defeats CockroachDB's replication and causes the system to be a single point of failure. Instead, start each node on a separate VM or machine.
* To start a node with multiple disks or SSDs, provide a separate `--store` flag for each disk when starting the `cockroach` process on the node. For more details about stores, see Start a Node.
If you start a node with multiple `--store` flags, it is not possible to scale back down to only using a single store on the node. Instead, you must decommission the node and start a new node with the updated `--store`.
* When starting each node, use the `--locality` flag to describe the node's location, for example, `--locality=region=west,zone=us-west-1`. The key-value pairs should be ordered from most to least inclusive, and the keys and order of key-value pairs must be the same on all nodes.
* When deploying in a single availability zone:
* To be able to tolerate the failure of any 1 node, use at least 3 nodes with the `default` 3-way replication factor. In this case, if 1 node fails, each range retains 2 of its 3 replicas, a majority.
* To be able to tolerate 2 simultaneous node failures, use at least 5 nodes and increase the `default` replication factor for user data to 5. The replication factor for important internal data is 5 by default, so no adjustments are needed for internal data. In this case, if 2 nodes fail at the same time, each range retains 3 of its 5 replicas, a majority.
* When deploying across multiple availability zones:
* To be able to tolerate the failure of 1 entire AZ in a region, use at least 3 AZs per region and set `--locality` on each node to spread data evenly across regions and AZs. In this case, if 1 AZ goes offline, the 2 remaining AZs retain a majority of replicas.
* To ensure that ranges are split evenly across nodes, use the same number of nodes in each AZ. This is to avoid overloading any nodes with excessive resource consumption.
* When deploying across multiple regions:
* To be able to tolerate the failure of 1 entire region, use at least 3 regions.
* This article covers the use of Linux instances with GCE. You may wish to review the instructions for [connecting to Windows instances](https://cloud.google.com/compute/docs/instances/connecting-to-instance#windows).
### Recommendations
* Decide how you want to access your DB Console:
| Access Level | Description |
| - | - |
| Partially open | Set a firewall rule to allow only specific IP addresses to communicate on port `8080`. |
| Completely open | Set a firewall rule to allow all IP addresses to communicate on port `8080`. |
| Completely closed | Set a firewall rule to disallow all communication on port `8080`. In this case, a machine with SSH access to a node could use an SSH tunnel to access the DB Console. |
## Step 1. Configure your network
CockroachDB requires TCP communication on two ports:
* `26257` for inter-node communication (i.e., working as a cluster)
* `8080` for exposing your DB Console
To expose your DB Console and allow traffic from the TCP proxy load balancer and health checker to your instances, [create firewall rules](https://cloud.google.com/compute/docs/vpc/firewalls) for your project. When creating firewall rules, we recommend using Google Cloud Platform's **tag** feature to apply the rule only to instances with the same tag.
#### DB Console
| Field | Recommended Value |
| - | - |
| Name | **cockroachadmin** |
| Source filter | IP ranges |
| Source IP ranges | Your local network's IP ranges |
| Allowed protocols... | **tcp:8080** |
| Target tags | **cockroachdb** |
#### Application data
Applications will not connect directly to your CockroachDB nodes. Instead, they'll connect to GCE's TCP Proxy Load Balancing service, which automatically routes traffic to the instances that are closest to the user. Because this service is implemented at the edge of the Google Cloud, you'll need to create a firewall rule to allow traffic from the load balancer and health checker to your instances. This is covered in [Step 4](#step-4-set-up-load-balancing).
## Step 2. Create instances
[Create an instance](https://cloud.google.com/compute/docs/instances/create-start-instance) for each node you plan to have in your cluster. If you plan to run a sample workload against the cluster, create a separate instance for that workload.
* Run at least 3 nodes to ensure survivability.
* Use general-purpose [`t2d-standard`, `n2-standard`, or `n2d-standard`](https://cloud.google.com/compute/pricing#predefined_machine_types) VMs, or use [custom VMs](https://cloud.google.com/compute/docs/instances/creating-instance-with-custom-machine-type). For example, Cockroach Labs has used `t2d-standard-8`, `n2-standard-8`, and `n2d-standard-8` for performance benchmarking.
Do not use `f1` or `g1` [shared-core machines](https://cloud.google.com/compute/docs/machine-types#sharedcore), which limit the load on CPU resources.
* If you used a tag for your firewall rules, when you create the instance, click **Management, security, disks, networking, sole tenancy**. Under the **Networking** tab, in the **Network tags** field, enter **cockroachdb**.
For more details, see Hardware Recommendations and Cluster Topology.
## Step 3. Synchronize clocks
CockroachDB requires moderate levels of clock synchronization to preserve data consistency. For this reason, when a node detects that its clock is out of sync with at least half of the other nodes in the cluster by 80% of the maximum offset allowed (500ms by default), it spontaneously shuts down. This avoids the risk of consistency anomalies, but it's best to prevent clocks from drifting too far in the first place by running clock synchronization software on each node.
[`ntpd`](http://doc.ntp.org/) should keep offsets in the single-digit milliseconds, so that software is featured here, but other methods of clock synchronization are suitable as well.
1. SSH to the first machine.
2. Disable `timesyncd`, which tends to be active by default on some Linux distributions:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo timedatectl set-ntp no
```
Verify that `timesyncd` is off:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ timedatectl
```
Look for `Network time on: no` or `NTP enabled: no` in the output.
3. Install the `ntp` package:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo apt-get install ntp
```
4. Stop the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp stop
```
5. Sync the machine's clock with Google's NTP service:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpd -b time.google.com
```
To make this change permanent, in the `/etc/ntp.conf` file, remove or comment out any lines starting with `server` or `pool` and add the following lines:
```
server time1.google.com iburst
server time2.google.com iburst
server time3.google.com iburst
server time4.google.com iburst
```
Restart the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp start
```
We recommend Google's NTP service because it handles ["smearing" the leap second](https://developers.google.com/time/smear). If you use a different NTP service that doesn't smear the leap second, be sure to configure client-side smearing in the same way on each machine. See the Production Checklist for details.
1. Verify that the machine is using a Google NTP server:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpq -p
```
The active NTP server will be marked with an asterisk.
2. Repeat these steps for each machine where a CockroachDB node will run.
Compute Engine instances are preconfigured to use [NTP](http://www.ntp.org/), which should keep offsets in the single-digit milliseconds. However, Google can’t predict how external NTP services, such as `pool.ntp.org`, will handle the leap second. Therefore, you should:
* [Configure each GCE instance to use Google's internal NTP service](https://cloud.google.com/compute/docs/instances/configure-ntp#configure_ntp_for_your_instances).
* If you plan to run a hybrid cluster across GCE and other cloud providers or environments, note that all of the nodes must be synced to the same time source, or to different sources that implement leap second smearing in the same way. See the Production Checklist for details.
Amazon provides the [Amazon Time Sync Service](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html), which uses a fleet of satellite-connected and atomic reference clocks in each AWS Region to deliver accurate current time readings. The service also smears the leap second.
* [Configure each AWS instance to use the internal Amazon Time Sync Service](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html#configure-amazon-time-service).
* Per the above instructions, ensure that `etc/chrony.conf` on the instance contains the line `server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4` and that other `server` or `pool` lines are commented out.
* To verify that Amazon Time Sync Service is being used, run `chronyc sources -v` and check for a line containing `* 169.254.169.123`. The `*` denotes the preferred time server.
* If you plan to run a hybrid cluster across GCE and other cloud providers or environments, note that all of the nodes must be synced to the same time source, or to different sources that implement leap second smearing in the same way. See the Production Checklist for details.
[`ntpd`](http://doc.ntp.org/) should keep offsets in the single-digit milliseconds, so that software is featured here. However, to run `ntpd` properly on Azure VMs, it's necessary to first unbind the Time Synchronization device used by the Hyper-V technology running Azure VMs; this device aims to synchronize time between the VM and its host operating system but has been known to cause problems.
1. SSH to the first machine.
2. Find the ID of the Hyper-V Time Synchronization device:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ curl -O https://raw.githubusercontent.com/torvalds/linux/master/tools/hv/lsvmbus
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ python lsvmbus -vv | grep -w "Time Synchronization" -A 3
```
```
VMBUS ID 12: Class_ID = {9527e630-d0ae-497b-adce-e80ab0175caf} - [Time Synchronization]
Device_ID = {2dd1ce17-079e-403c-b352-a1921ee207ee}
Sysfs path: /sys/bus/vmbus/devices/2dd1ce17-079e-403c-b352-a1921ee207ee
Rel_ID=12, target_cpu=0
```
3. Unbind the device, using the `Device_ID` from the previous command's output:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ echo | sudo tee /sys/bus/vmbus/drivers/hv_utils/unbind
```
4. Install the `ntp` package:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo apt-get install ntp
```
5. Stop the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp stop
```
6. Sync the machine's clock with Google's NTP service:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpd -b time.google.com
```
To make this change permanent, in the `/etc/ntp.conf` file, remove or comment out any lines starting with `server` or `pool` and add the following lines:
```
server time1.google.com iburst
server time2.google.com iburst
server time3.google.com iburst
server time4.google.com iburst
```
Restart the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp start
```
We recommend Google's NTP service because it handles ["smearing" the leap second](https://developers.google.com/time/smear). If you use a different NTP service that doesn't smear the leap second, be sure to configure client-side smearing in the same way on each machine. See the Production Checklist for details.
7. Verify that the machine is using a Google NTP server:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpq -p
```
The active NTP server will be marked with an asterisk.
8. Repeat these steps for each machine where a CockroachDB node will run.
## Step 4. Set up load balancing
Each CockroachDB node is an equally suitable SQL gateway to your cluster, but to ensure client performance and reliability, it's important to use load balancing:
* **Performance:** Load balancers spread client traffic across nodes. This prevents any one node from being overwhelmed by requests and improves overall cluster performance (queries per second).
* **Reliability:** Load balancers decouple client health from the health of a single CockroachDB node. In cases where a node fails, the load balancer redirects client traffic to available nodes.
GCE offers fully-managed [TCP Proxy Load Balancing](https://cloud.google.com/load-balancing/docs/tcp/). This service lets you use a single IP address for all users around the world, automatically routing traffic to the instances that are closest to the user.
When using TCP Proxy Load Balancing, you cannot use firewall rules to control access to the load balancer. If you need such control, consider using [Network TCP Load Balancing](https://cloud.google.com/compute/docs/load-balancing/network/) instead, but note that it cannot be used across regions. You might also consider using the HAProxy load balancer (see the On-Premises tutorial for guidance).
To use GCE's TCP Proxy Load Balancing service:
1. For each zone in which you're running an instance, [create a distinct instance group](https://cloud.google.com/compute/docs/instance-groups/creating-groups-of-unmanaged-instances).
* To ensure that the load balancer knows where to direct traffic, specify a port name mapping, with `tcp26257` as the **Port name** and `26257` as the **Port number**.
2. [Add the relevant instances to each instance group](https://cloud.google.com/compute/docs/instance-groups/creating-groups-of-unmanaged-instances#addinstances).
3. [Configure TCP Proxy Load Balancing](https://cloud.google.com/load-balancing/docs/tcp/setting-up-tcp#configure_load_balancer).
* During backend configuration, create a health check, setting the **Protocol** to `HTTPS`, the **Port** to `8080`, and the **Request path** to path `/health?ready=1`. This health endpoint ensures that load balancers do not direct traffic to nodes that are live but not ready to receive requests.
* If you want to maintain long-lived SQL connections that may be idle for more than tens of seconds, increase the backend timeout setting accordingly.
* During frontend configuration, reserve a static IP address and note the IP address and the port you select. You'll use this address and port for all client connections.
4. [Create a firewall rule](https://cloud.google.com/load-balancing/docs/tcp/setting-up-tcp#config-hc-firewall) to allow traffic from the load balancer and health checker to your instances. This is necessary because TCP Proxy Load Balancing is implemented at the edge of the Google Cloud.
* Be sure to set **Source IP ranges** to `130.211.0.0/22` and `35.191.0.0/16` and set **Target tags** to `cockroachdb` (not to the value specified in the linked instructions).
## Step 5. Generate certificates
You can use `cockroach cert` commands or `openssl` commands to generate security certificates. This section features the `cockroach cert` commands.
Locally, you'll need to create the following certificates and keys:
* A certificate authority (CA) key pair (`ca.crt` and `ca.key`).
* A node key pair for each node, issued to its IP addresses and any common names the machine uses, as well as to the IP addresses and common names for machines running load balancers.
* A client key pair for the `root` user. You'll use this to run a sample workload against the cluster as well as some `cockroach` client commands from your local machine.
1. Install CockroachDB on your local machine, if you haven't already.
2. Create two directories:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir certs
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir my-safe-directory
```
* `certs`: You'll generate your CA certificate and all node and client certificates and keys in this directory and then upload some of the files to your nodes.
* `my-safe-directory`: You'll generate your CA key in this directory and then reference the key when generating node and client certificates. After that, you'll keep the key safe and secret; you will not upload it to your nodes.
3. Create the CA certificate and key:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-ca \
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
4. Create the certificate and key for the first node, issued to all common names you might use to refer to the node as well as to the load balancer instances:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-node \
\
\
\
\
localhost \
127.0.0.1 \
\
\
\
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
5. Upload the CA certificate and node certificate and key to the first node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ gcloud compute ssh \
--project \
--command "mkdir certs"
```
`gcloud compute ssh` associates your public SSH key with the GCP project and is only needed when connecting to the first node. See the [GCP docs](https://cloud.google.com/sdk/gcloud/reference/compute/ssh) for more details.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ scp certs/ca.crt \
certs/node.crt \
certs/node.key \
@@@@@
This is necessary because the certificates and keys for additional nodes will also be named `node.crt` and `node.key`. As an alternative to deleting these files, you can run the next `cockroach cert create-node` commands with the `--overwrite` flag.
1. Create the certificate and key for the second node, issued to all common names you might use to refer to the node as well as to the load balancer instances:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-node \
\
\
\
\
localhost \
127.0.0.1 \
\
\
\
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
2. Upload the CA certificate and node certificate and key to the second node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ ssh @@@@@@`cockroach` client commands from your local machine. If you might also want to run `cockroach` client commands directly on a node (e.g., for local debugging), you'll need to copy the `root` user's certificate and key to that node as well.
On accessing the DB Console in a later step, your browser will consider the CockroachDB-created certificate invalid and you’ll need to click through a warning message to get to the UI. You can avoid this issue by using a certificate issued by a public CA.
## Step 6. Start nodes
By default, inter-node communication uses the internal IP addresses of your GCE instances.
You can start the nodes manually or automate the process using [systemd](https://www.freedesktop.org/wiki/Software/systemd/).
For each initial node of your cluster, complete the following steps:
After completing these steps, nodes will not yet be live. They will complete the startup process and join together to form a cluster as soon as the cluster is initialized in the next step.
1. Visit Releases and download the full binary of CockroachDB to the node.
2. On the node, follow the instructions to install CockroachDB.
3. Run the `cockroach start` command:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--advertise-addr=
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
| `--cache`
`--max-sql-memory` | Increases the node's cache size to 25% of available system memory to improve read performance. The capacity for in-memory SQL processing defaults to 25% of system memory but can be raised, if necessary, to increase the number of simultaneous client connections allowed by the node as well as the node's capacity for in-memory processing of rows when using `ORDER BY`, `GROUP BY`, `DISTINCT`, joins, and window functions. For more details, see Cache and SQL Memory Size. |
| `--background` | Starts the node in the background so you gain control of the terminal to issue more commands. |
When deploying across multiple datacenters, or when there is otherwise high latency between nodes, it is recommended to set `--locality` as well. For more details, see Locality.
For other flags not explicitly set, the command uses default values. For example, the node stores data in `--store=cockroach-data` and binds DB Console HTTP requests to `--http-addr=Start a Node.
Repeat these steps for each additional node that you want in your cluster.
For each initial node of your cluster, complete the following steps:
After completing these steps, nodes will not yet be live. They will complete the startup process and join together to form a cluster as soon as the cluster is initialized in the next step.
1. SSH to the machine where you want the node to run. Ensure you are logged in as the `root` user.
2. Install CockroachDB for Linux.
3. Create the Cockroach directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir /var/lib/cockroach
```
4. Create a Unix user named `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ useradd cockroach
```
5. Move the `certs` directory to the `cockroach` directory.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mv certs /var/lib/cockroach/
```
6. Change the ownership of the `cockroach` directory to the user `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ chown -R cockroach /var/lib/cockroach
```
7. Download the [sample configuration template](https://raw.githubusercontent.com/cockroachlabs/docs/main/src/current/_includes/v26.3/prod-deployment/securecockroachdb.service) and save the file in the `/etc/systemd/system/` directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl -o securecockroachdb.service https://raw.githubusercontent.com/cockroachlabs/docs/main/src/current/_includes/v23.2/prod-deployment/securecockroachdb.service
```
Alternatively, you can create the file yourself and copy the script into it:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
[Unit]
Description=Cockroach Database cluster node
Requires=network.target
[Service]
Type=notify
WorkingDirectory=/var/lib/cockroach
ExecStart=/usr/local/bin/cockroach start --certs-dir=certs --advertise-addr=\
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
When deploying across multiple datacenters, or when there is otherwise high latency between nodes, it is recommended to set `--locality` as well. For more details, see Locality.
For other flags not explicitly set, the command uses default values. For example, the node stores data in `--store=cockroach-data` and binds DB Console HTTP requests to `--http-addr=localhost:8080`. To set these options manually, see Start a Node.
9. Start the CockroachDB cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
systemctl start securecockroachdb
```
10. Configure `systemd` to start CockroachDB automatically after a reboot:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
systemctl enable securecockroachdb
```
11. Repeat these steps for each additional node that you want in your cluster.
`systemd` handles node restarts in case of node failure. To stop a node without `systemd` restarting it, run `systemctl stop securecockroachdb`
## Step 7. Initialize the cluster
On your local machine, run the `cockroach init` command to complete the node startup process and have them join together as a cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach init --certs-dir=certs --host=standard output, such as the CockroachDB version, the URL for the DB Console, and the SQL URL for clients.
## Step 8. Test the cluster
CockroachDB replicates and distributes data behind-the-scenes and uses a [Gossip protocol](https://wikipedia.org/wiki/Gossip_protocol) to enable each node to locate data across the cluster. Once a cluster is live, any node can be used as a SQL gateway.
When using a load balancer, you should issue commands directly to the load balancer, which then routes traffic to the nodes.
Use the built-in SQL client locally as follows:
1. On your local machine, launch the built-in SQL client, with the `--host` flag set to the address of the load balancer:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --certs-dir=certs --host= CREATE DATABASE securenodetest;
```
3. View the cluster's databases, which will include `securenodetest`:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SHOW DATABASES;
```
```
+--------------------+
| Database |
+--------------------+
| crdb_internal |
| information_schema |
| securenodetest |
| pg_catalog |
| system |
+--------------------+
(5 rows)
```
4. Use `\q` to exit the SQL shell.
## Step 9. Run a sample workload
CockroachDB comes with a number of built-in workloads for simulating client traffic. This step features CockroachDB's version of the [TPC-C](http://www.tpc.org/tpcc/) workload.
Be sure that you have configured your network to allow traffic from the application to the load balancer. In this case, you will run the sample workload on one of your machines. The traffic source should therefore be the **internal (private)** IP address of that machine.
1. SSH to the machine where you want to run the sample TPC-C workload.
This should be a machine that is not running a CockroachDB node, and it should already have a `certs` directory containing `ca.crt`, `client.root.crt`, and `client.root.key` files.
2. Install CockroachDB for Linux.
3. Use the `cockroach workload` command to load the initial schema and data, pointing it at the IP address of the load balancer:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload init tpcc \
'postgresql://root@:26257/tpcc?sslmode=verify-full&sslrootcert=certs/ca.crt&sslcert=certs/client.root.crt&sslkey=certs/client.root.key'
```
4. Use the `cockroach workload` command to run the workload for 10 minutes:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload run tpcc \
--duration=10m \
'postgresql://root@:26257/tpcc?sslmode=verify-full&sslrootcert=certs/ca.crt&sslcert=certs/client.root.crt&sslkey=certs/client.root.key'
```
You'll see per-operation statistics print to standard output every second:
```
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
1s 0 1443.4 1494.8 4.7 9.4 27.3 67.1 transfer
2s 0 1686.5 1590.9 4.7 8.1 15.2 28.3 transfer
3s 0 1735.7 1639.0 4.7 7.3 11.5 28.3 transfer
4s 0 1542.6 1614.9 5.0 8.9 12.1 21.0 transfer
5s 0 1695.9 1631.1 4.7 7.3 11.5 22.0 transfer
6s 0 1569.2 1620.8 5.0 8.4 11.5 15.7 transfer
7s 0 1614.6 1619.9 4.7 8.1 12.1 16.8 transfer
8s 0 1344.4 1585.6 5.8 10.0 15.2 31.5 transfer
9s 0 1351.9 1559.5 5.8 10.0 16.8 54.5 transfer
10s 0 1514.8 1555.0 5.2 8.1 12.1 16.8 transfer
...
```
After the specified duration (10 minutes in this case), the workload will stop and you'll see totals printed to standard output:
```
_elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__result
600.0s 0 823902 1373.2 5.8 5.5 10.0 15.2 209.7
```
For more `tpcc` options, use `cockroach workload run tpcc --help`. For details about other workloads built into the `cockroach` binary, use `cockroach workload --help`.
1. To monitor the load generator's progress, open the DB Console by pointing a browser to the address in the `admin` field in the standard output of any node on startup.
Since the load generator is pointed at the load balancer, the connections will be evenly distributed across nodes. To verify this, click **Metrics** on the left, select the **SQL** dashboard, and then check the **SQL Connections** graph. You can use the **Graph** menu to filter the graph for specific nodes.
## Step 10. Monitor the cluster
Despite CockroachDB's various built-in safeguards against failure, it is critical to actively monitor the overall health and performance of a cluster running in production and to create alerting rules that promptly send notifications when there are events that require investigation or intervention.
For details about available monitoring options and the most important events and metrics to alert on, see Monitoring and Alerting.
## Step 11. Scale the cluster
You can start the nodes manually or automate the process using [systemd](https://www.freedesktop.org/wiki/Software/systemd/).
For each additional node you want to add to the cluster, complete the following steps:
1. SSH to the machine where you want the node to run.
2. Install CockroachDB for Linux.
3. Run the `cockroach start` command, passing the new node's address as the `--advertise-addr` flag and pointing `--join` to the three existing nodes (also include `--locality` if you set it earlier).
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--advertise-addr=
For each additional node you want to add to the cluster, complete the following steps:
1. SSH to the machine where you want the node to run. Ensure you are logged in as the `root` user.
2. Install CockroachDB for Linux.
3. Create the Cockroach directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
mkdir /var/lib/cockroach
```
4. Create a Unix user named `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
useradd cockroach
```
5. Move the `certs` directory to the `cockroach` directory.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
mv certs /var/lib/cockroach/
```
6. Change the ownership of the `cockroach` directory to the user `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
chown -R cockroach /var/lib/cockroach
```
7. Download the [sample configuration template](https://raw.githubusercontent.com/cockroachlabs/docs/master/_includes/v26.3/prod-deployment/securecockroachdb.service):
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl -o securecockroachdb.service https://raw.githubusercontent.com/cockroachlabs/docs/master/_includes/v26.3/prod-deployment/securecockroachdb.service
```
Alternatively, you can create the file yourself and copy the script into it:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
[Unit]
Description=Cockroach Database cluster node
Requires=network.target
[Service]
Type=notify
WorkingDirectory=/var/lib/cockroach
ExecStart=/usr/local/bin/cockroach start --certs-dir=certs --advertise-addr=\
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
9. Repeat these steps for each additional node that you want in your cluster.
## Step 12. Use the database
Now that your deployment is working, you can:
1. Implement your data model.
2. Create users and grant them privileges.
3. Connect your application. Be sure to connect your application to the load balancer, not to a CockroachDB node.
4. Take backups of your data.
You may also want to adjust the way the cluster replicates data. For example, by default, a multi-node cluster replicates all data 3 times; you can change this replication factor or create additional rules for replicating individual databases and tables differently. For more information, see Replication Controls.
When running a cluster of 5 nodes or more, it's safest to increase the replication factor for important internal data to 5, even if you do not do so for user data. For the cluster as a whole to remain available, the ranges for this internal data must always retain a majority of their replicas.
## See also
* Production Checklist
* Manual Deployment
* Orchestrated Deployment
* Monitoring and Alerting
* Performance Tuning
* Local Deployment
# Deploy CockroachDB on Microsoft Azure
Source: https://docs.cockroachlabs.com/docs/v26.3/deploy-cockroachdb-on-microsoft-azure
This page shows you how to manually deploy a secure multi-node CockroachDB cluster on Microsoft Azure, using Azure's managed load balancing service to distribute client traffic.
If you are only testing CockroachDB, or you are not concerned with protecting network communication with TLS encryption, you can use an insecure cluster instead. Select **Insecure** above for instructions.
To try CockroachDB Cloud instead of running CockroachDB yourself, refer to the Cloud Quickstart.
## Before you begin
### Requirements
* You must have CockroachDB installed locally. This is necessary for generating and managing your deployment's certificates.
* You must have [SSH access]() to each machine. This is necessary for distributing and starting CockroachDB binaries.
* Your network configuration must allow TCP communication on the following ports:
* `26257` for intra-cluster and client-cluster communication
* `8080` to expose your DB Console
* Carefully review the Production Checklist, including supported hardware and software, and the recommended Topology Patterns.
* Do not run multiple node processes on the same VM or machine. This defeats CockroachDB's replication and causes the system to be a single point of failure. Instead, start each node on a separate VM or machine.
* To start a node with multiple disks or SSDs, provide a separate `--store` flag for each disk when starting the `cockroach` process on the node. For more details about stores, see Start a Node.
If you start a node with multiple `--store` flags, it is not possible to scale back down to only using a single store on the node. Instead, you must decommission the node and start a new node with the updated `--store`.
* When starting each node, use the `--locality` flag to describe the node's location, for example, `--locality=region=west,zone=us-west-1`. The key-value pairs should be ordered from most to least inclusive, and the keys and order of key-value pairs must be the same on all nodes.
* When deploying in a single availability zone:
* To be able to tolerate the failure of any 1 node, use at least 3 nodes with the `default` 3-way replication factor. In this case, if 1 node fails, each range retains 2 of its 3 replicas, a majority.
* To be able to tolerate 2 simultaneous node failures, use at least 5 nodes and increase the `default` replication factor for user data to 5. The replication factor for important internal data is 5 by default, so no adjustments are needed for internal data. In this case, if 2 nodes fail at the same time, each range retains 3 of its 5 replicas, a majority.
* When deploying across multiple availability zones:
* To be able to tolerate the failure of 1 entire AZ in a region, use at least 3 AZs per region and set `--locality` on each node to spread data evenly across regions and AZs. In this case, if 1 AZ goes offline, the 2 remaining AZs retain a majority of replicas.
* To ensure that ranges are split evenly across nodes, use the same number of nodes in each AZ. This is to avoid overloading any nodes with excessive resource consumption.
* When deploying across multiple regions:
* To be able to tolerate the failure of 1 entire region, use at least 3 regions.
### Recommendations
* Decide how you want to access your DB Console:
| Access Level | Description |
| - | - |
| Partially open | Set a firewall rule to allow only specific IP addresses to communicate on port `8080`. |
| Completely open | Set a firewall rule to allow all IP addresses to communicate on port `8080`. |
| Completely closed | Set a firewall rule to disallow all communication on port `8080`. In this case, a machine with SSH access to a node could use an SSH tunnel to access the DB Console. |
## Step 1. Configure your network
CockroachDB requires TCP communication on two ports:
* **26257** (`tcp:26257`) for inter-node communication (i.e., working as a cluster), for applications to connect to the load balancer, and for routing from the load balancer to nodes
* **8080** (`tcp:8080`) for exposing your DB Console
To enable this in Azure, you must create a Resource Group, Virtual Network, and Network Security Group.
1. [Create a Resource Group](https://azure.microsoft.com/updates/create-empty-resource-groups/).
2. [Create a Virtual Network](https://docs.microsoft.com/azure/virtual-network/virtual-networks-create-vnet-arm-pportal) that uses your **Resource Group**.
3. [Create a Network Security Group](https://docs.microsoft.com/azure/virtual-network/virtual-networks-create-nsg-arm-pportal) that uses your **Resource Group**, and then add the following **inbound** rules to it:
* **DB Console support**:
| Field | Recommended Value |
| - | - |
| Name | **cockroachadmin** |
| Source | **IP Addresses** |
| Source IP addresses/CIDR ranges | Your local network’s IP ranges |
| Source port ranges | \* |
| Destination | **Any** |
| Destination port range | **8080** |
| Protocol | **TCP** |
| Action | **Allow** |
| Priority | Any value > 1000 |
* **Application support**:
| Field | Recommended Value |
| - | - |
| Name | **cockroachapp** |
| Source | **IP Addresses** |
| Source IP addresses/CIDR ranges | Your local network’s IP ranges |
| Source port ranges | \* |
| Destination | **Any** |
| Destination port range | **26257** |
| Protocol | **TCP** |
| Action | **Allow** |
| Priority | Any value > 1000 |
## Step 2. Create VMs
[Create Linux VMs](https://docs.microsoft.com/azure/virtual-machine-scale-sets/quick-create-portal) for each node you plan to have in your cluster. If you plan to run a sample workload against the cluster, create a separate VM for that workload.
* Run at least 3 nodes to ensure survivability.
* Use general-purpose [Dsv5-series](https://docs.microsoft.com/azure/virtual-machines/dv5-dsv5-series) and [Dasv5-series](https://docs.microsoft.com/azure/virtual-machines/dasv5-dadsv5-series) or memory-optimized [Ev5-series](https://docs.microsoft.com/azure/virtual-machines/ev5-esv5-series) and [Easv5-series](https://docs.microsoft.com/azure/virtual-machines/easv5-eadsv5-series#easv5-series) VMs. For example, Cockroach Labs has used `Standard_D8s_v5`, `Standard_D8as_v5`, `Standard_E8s_v5`, and `Standard_e8as_v5` for performance benchmarking.
* Compute-optimized [F-series](https://docs.microsoft.com/azure/virtual-machines/fsv2-series) VMs are also acceptable.
Do not use ["burstable" B-series](https://docs.microsoft.com/azure/virtual-machines/linux/b-series-burstable) VMs, which limit the load on CPU resources. Also, Cockroach Labs has experienced data corruption issues on A-series VMs, so we recommend avoiding those as well.
* When creating the VMs, make sure to select the **Resource Group**, **Virtual Network**, and **Network Security Group** you created.
For more details, see Hardware Recommendations and Cluster Topology.
## Step 3. Synchronize clocks
CockroachDB requires moderate levels of clock synchronization to preserve data consistency. For this reason, when a node detects that its clock is out of sync with at least half of the other nodes in the cluster by 80% of the maximum offset allowed (500ms by default), it spontaneously shuts down. This avoids the risk of consistency anomalies, but it's best to prevent clocks from drifting too far in the first place by running clock synchronization software on each node.
[`ntpd`](http://doc.ntp.org/) should keep offsets in the single-digit milliseconds, so that software is featured here, but other methods of clock synchronization are suitable as well.
1. SSH to the first machine.
2. Disable `timesyncd`, which tends to be active by default on some Linux distributions:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo timedatectl set-ntp no
```
Verify that `timesyncd` is off:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ timedatectl
```
Look for `Network time on: no` or `NTP enabled: no` in the output.
3. Install the `ntp` package:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo apt-get install ntp
```
4. Stop the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp stop
```
5. Sync the machine's clock with Google's NTP service:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpd -b time.google.com
```
To make this change permanent, in the `/etc/ntp.conf` file, remove or comment out any lines starting with `server` or `pool` and add the following lines:
```
server time1.google.com iburst
server time2.google.com iburst
server time3.google.com iburst
server time4.google.com iburst
```
Restart the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp start
```
We recommend Google's NTP service because it handles ["smearing" the leap second](https://developers.google.com/time/smear). If you use a different NTP service that doesn't smear the leap second, be sure to configure client-side smearing in the same way on each machine. See the Production Checklist for details.
1. Verify that the machine is using a Google NTP server:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpq -p
```
The active NTP server will be marked with an asterisk.
2. Repeat these steps for each machine where a CockroachDB node will run.
Compute Engine instances are preconfigured to use [NTP](http://www.ntp.org/), which should keep offsets in the single-digit milliseconds. However, Google can’t predict how external NTP services, such as `pool.ntp.org`, will handle the leap second. Therefore, you should:
* [Configure each GCE instance to use Google's internal NTP service](https://cloud.google.com/compute/docs/instances/configure-ntp#configure_ntp_for_your_instances).
* If you plan to run a hybrid cluster across GCE and other cloud providers or environments, note that all of the nodes must be synced to the same time source, or to different sources that implement leap second smearing in the same way. See the Production Checklist for details.
Amazon provides the [Amazon Time Sync Service](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html), which uses a fleet of satellite-connected and atomic reference clocks in each AWS Region to deliver accurate current time readings. The service also smears the leap second.
* [Configure each AWS instance to use the internal Amazon Time Sync Service](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html#configure-amazon-time-service).
* Per the above instructions, ensure that `etc/chrony.conf` on the instance contains the line `server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4` and that other `server` or `pool` lines are commented out.
* To verify that Amazon Time Sync Service is being used, run `chronyc sources -v` and check for a line containing `* 169.254.169.123`. The `*` denotes the preferred time server.
* If you plan to run a hybrid cluster across GCE and other cloud providers or environments, note that all of the nodes must be synced to the same time source, or to different sources that implement leap second smearing in the same way. See the Production Checklist for details.
[`ntpd`](http://doc.ntp.org/) should keep offsets in the single-digit milliseconds, so that software is featured here. However, to run `ntpd` properly on Azure VMs, it's necessary to first unbind the Time Synchronization device used by the Hyper-V technology running Azure VMs; this device aims to synchronize time between the VM and its host operating system but has been known to cause problems.
1. SSH to the first machine.
2. Find the ID of the Hyper-V Time Synchronization device:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ curl -O https://raw.githubusercontent.com/torvalds/linux/master/tools/hv/lsvmbus
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ python lsvmbus -vv | grep -w "Time Synchronization" -A 3
```
```
VMBUS ID 12: Class_ID = {9527e630-d0ae-497b-adce-e80ab0175caf} - [Time Synchronization]
Device_ID = {2dd1ce17-079e-403c-b352-a1921ee207ee}
Sysfs path: /sys/bus/vmbus/devices/2dd1ce17-079e-403c-b352-a1921ee207ee
Rel_ID=12, target_cpu=0
```
3. Unbind the device, using the `Device_ID` from the previous command's output:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ echo | sudo tee /sys/bus/vmbus/drivers/hv_utils/unbind
```
4. Install the `ntp` package:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo apt-get install ntp
```
5. Stop the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp stop
```
6. Sync the machine's clock with Google's NTP service:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpd -b time.google.com
```
To make this change permanent, in the `/etc/ntp.conf` file, remove or comment out any lines starting with `server` or `pool` and add the following lines:
```
server time1.google.com iburst
server time2.google.com iburst
server time3.google.com iburst
server time4.google.com iburst
```
Restart the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp start
```
We recommend Google's NTP service because it handles ["smearing" the leap second](https://developers.google.com/time/smear). If you use a different NTP service that doesn't smear the leap second, be sure to configure client-side smearing in the same way on each machine. See the Production Checklist for details.
7. Verify that the machine is using a Google NTP server:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpq -p
```
The active NTP server will be marked with an asterisk.
8. Repeat these steps for each machine where a CockroachDB node will run.
## Step 4. Set up load balancing
Each CockroachDB node is an equally suitable SQL gateway to your cluster, but to ensure client performance and reliability, it's important to use load balancing:
* **Performance:** Load balancers spread client traffic across nodes. This prevents any one node from being overwhelmed by requests and improves overall cluster performance (queries per second).
* **Reliability:** Load balancers decouple client health from the health of a single CockroachDB node. In cases where a node fails, the load balancer redirects client traffic to available nodes.
Microsoft Azure offers fully-managed load balancing to distribute traffic between instances.
1. [Add Azure load balancing](https://docs.microsoft.com/azure/load-balancer/load-balancer-overview). Be sure to:
* Set forwarding rules to route TCP traffic from the load balancer's port **26257** to port **26257** on the nodes.
* Configure health checks to use HTTP port **8080** and path `/health?ready=1`. This health endpoint ensures that load balancers do not direct traffic to nodes that are live but not ready to receive requests.
1. Note the provisioned **IP Address** for the load balancer. You'll use this later to test load balancing and to connect your application to the cluster.
## Step 5. Generate certificates
You can use `cockroach cert` commands or `openssl` commands to generate security certificates. This section features the `cockroach cert` commands.
Locally, you'll need to create the following certificates and keys:
* A certificate authority (CA) key pair (`ca.crt` and `ca.key`).
* A node key pair for each node, issued to its IP addresses and any common names the machine uses, as well as to the IP addresses and common names for machines running load balancers.
* A client key pair for the `root` user. You'll use this to run a sample workload against the cluster as well as some `cockroach` client commands from your local machine.
1. Install CockroachDB on your local machine, if you haven't already.
2. Create two directories:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir certs
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir my-safe-directory
```
* `certs`: You'll generate your CA certificate and all node and client certificates and keys in this directory and then upload some of the files to your nodes.
* `my-safe-directory`: You'll generate your CA key in this directory and then reference the key when generating node and client certificates. After that, you'll keep the key safe and secret; you will not upload it to your nodes.
3. Create the CA certificate and key:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-ca \
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
4. Create the certificate and key for the first node, issued to all common names you might use to refer to the node as well as to the load balancer instances:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-node \
\
\
\
\
localhost \
127.0.0.1 \
\
\
\
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
5. Upload the CA certificate and node certificate and key to the first node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ gcloud compute ssh \
--project \
--command "mkdir certs"
```
`gcloud compute ssh` associates your public SSH key with the GCP project and is only needed when connecting to the first node. See the [GCP docs](https://cloud.google.com/sdk/gcloud/reference/compute/ssh) for more details.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ scp certs/ca.crt \
certs/node.crt \
certs/node.key \
@@@@@
This is necessary because the certificates and keys for additional nodes will also be named `node.crt` and `node.key`. As an alternative to deleting these files, you can run the next `cockroach cert create-node` commands with the `--overwrite` flag.
1. Create the certificate and key for the second node, issued to all common names you might use to refer to the node as well as to the load balancer instances:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-node \
\
\
\
\
localhost \
127.0.0.1 \
\
\
\
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
2. Upload the CA certificate and node certificate and key to the second node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ ssh @@@@@@`cockroach` client commands from your local machine. If you might also want to run `cockroach` client commands directly on a node (e.g., for local debugging), you'll need to copy the `root` user's certificate and key to that node as well.
On accessing the DB Console in a later step, your browser will consider the CockroachDB-created certificate invalid and you’ll need to click through a warning message to get to the UI. You can avoid this issue by using a certificate issued by a public CA.
## Step 6. Start nodes
You can start the nodes manually or automate the process using [systemd](https://www.freedesktop.org/wiki/Software/systemd/).
For each initial node of your cluster, complete the following steps:
After completing these steps, nodes will not yet be live. They will complete the startup process and join together to form a cluster as soon as the cluster is initialized in the next step.
1. Visit Releases and download the full binary of CockroachDB to the node.
2. On the node, follow the instructions to install CockroachDB.
3. Run the `cockroach start` command:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--advertise-addr=
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
| `--cache`
`--max-sql-memory` | Increases the node's cache size to 25% of available system memory to improve read performance. The capacity for in-memory SQL processing defaults to 25% of system memory but can be raised, if necessary, to increase the number of simultaneous client connections allowed by the node as well as the node's capacity for in-memory processing of rows when using `ORDER BY`, `GROUP BY`, `DISTINCT`, joins, and window functions. For more details, see Cache and SQL Memory Size. |
| `--background` | Starts the node in the background so you gain control of the terminal to issue more commands. |
When deploying across multiple datacenters, or when there is otherwise high latency between nodes, it is recommended to set `--locality` as well. For more details, see Locality.
For other flags not explicitly set, the command uses default values. For example, the node stores data in `--store=cockroach-data` and binds DB Console HTTP requests to `--http-addr=Start a Node.
Repeat these steps for each additional node that you want in your cluster.
For each initial node of your cluster, complete the following steps:
After completing these steps, nodes will not yet be live. They will complete the startup process and join together to form a cluster as soon as the cluster is initialized in the next step.
1. SSH to the machine where you want the node to run. Ensure you are logged in as the `root` user.
2. Install CockroachDB for Linux.
3. Create the Cockroach directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir /var/lib/cockroach
```
4. Create a Unix user named `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ useradd cockroach
```
5. Move the `certs` directory to the `cockroach` directory.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mv certs /var/lib/cockroach/
```
6. Change the ownership of the `cockroach` directory to the user `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ chown -R cockroach /var/lib/cockroach
```
7. Download the [sample configuration template](https://raw.githubusercontent.com/cockroachlabs/docs/main/src/current/_includes/v26.3/prod-deployment/securecockroachdb.service) and save the file in the `/etc/systemd/system/` directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl -o securecockroachdb.service https://raw.githubusercontent.com/cockroachlabs/docs/main/src/current/_includes/v23.2/prod-deployment/securecockroachdb.service
```
Alternatively, you can create the file yourself and copy the script into it:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
[Unit]
Description=Cockroach Database cluster node
Requires=network.target
[Service]
Type=notify
WorkingDirectory=/var/lib/cockroach
ExecStart=/usr/local/bin/cockroach start --certs-dir=certs --advertise-addr=\
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
When deploying across multiple datacenters, or when there is otherwise high latency between nodes, it is recommended to set `--locality` as well. For more details, see Locality.
For other flags not explicitly set, the command uses default values. For example, the node stores data in `--store=cockroach-data` and binds DB Console HTTP requests to `--http-addr=localhost:8080`. To set these options manually, see Start a Node.
9. Start the CockroachDB cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
systemctl start securecockroachdb
```
10. Configure `systemd` to start CockroachDB automatically after a reboot:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
systemctl enable securecockroachdb
```
11. Repeat these steps for each additional node that you want in your cluster.
`systemd` handles node restarts in case of node failure. To stop a node without `systemd` restarting it, run `systemctl stop securecockroachdb`
## Step 7. Initialize the cluster
On your local machine, run the `cockroach init` command to complete the node startup process and have them join together as a cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach init --certs-dir=certs --host=standard output, such as the CockroachDB version, the URL for the DB Console, and the SQL URL for clients.
## Step 8. Test the cluster
CockroachDB replicates and distributes data behind-the-scenes and uses a [Gossip protocol](https://wikipedia.org/wiki/Gossip_protocol) to enable each node to locate data across the cluster. Once a cluster is live, any node can be used as a SQL gateway.
When using a load balancer, you should issue commands directly to the load balancer, which then routes traffic to the nodes.
Use the built-in SQL client locally as follows:
1. On your local machine, launch the built-in SQL client, with the `--host` flag set to the address of the load balancer:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --certs-dir=certs --host= CREATE DATABASE securenodetest;
```
3. View the cluster's databases, which will include `securenodetest`:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SHOW DATABASES;
```
```
+--------------------+
| Database |
+--------------------+
| crdb_internal |
| information_schema |
| securenodetest |
| pg_catalog |
| system |
+--------------------+
(5 rows)
```
4. Use `\q` to exit the SQL shell.
## Step 9. Run a sample workload
CockroachDB comes with a number of built-in workloads for simulating client traffic. This step features CockroachDB's version of the [TPC-C](http://www.tpc.org/tpcc/) workload.
Be sure that you have configured your network to allow traffic from the application to the load balancer. In this case, you will run the sample workload on one of your machines. The traffic source should therefore be the **internal (private)** IP address of that machine.
1. SSH to the machine where you want to run the sample TPC-C workload.
This should be a machine that is not running a CockroachDB node, and it should already have a `certs` directory containing `ca.crt`, `client.root.crt`, and `client.root.key` files.
2. Install CockroachDB for Linux.
3. Use the `cockroach workload` command to load the initial schema and data, pointing it at the IP address of the load balancer:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload init tpcc \
'postgresql://root@:26257/tpcc?sslmode=verify-full&sslrootcert=certs/ca.crt&sslcert=certs/client.root.crt&sslkey=certs/client.root.key'
```
4. Use the `cockroach workload` command to run the workload for 10 minutes:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload run tpcc \
--duration=10m \
'postgresql://root@:26257/tpcc?sslmode=verify-full&sslrootcert=certs/ca.crt&sslcert=certs/client.root.crt&sslkey=certs/client.root.key'
```
You'll see per-operation statistics print to standard output every second:
```
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
1s 0 1443.4 1494.8 4.7 9.4 27.3 67.1 transfer
2s 0 1686.5 1590.9 4.7 8.1 15.2 28.3 transfer
3s 0 1735.7 1639.0 4.7 7.3 11.5 28.3 transfer
4s 0 1542.6 1614.9 5.0 8.9 12.1 21.0 transfer
5s 0 1695.9 1631.1 4.7 7.3 11.5 22.0 transfer
6s 0 1569.2 1620.8 5.0 8.4 11.5 15.7 transfer
7s 0 1614.6 1619.9 4.7 8.1 12.1 16.8 transfer
8s 0 1344.4 1585.6 5.8 10.0 15.2 31.5 transfer
9s 0 1351.9 1559.5 5.8 10.0 16.8 54.5 transfer
10s 0 1514.8 1555.0 5.2 8.1 12.1 16.8 transfer
...
```
After the specified duration (10 minutes in this case), the workload will stop and you'll see totals printed to standard output:
```
_elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__result
600.0s 0 823902 1373.2 5.8 5.5 10.0 15.2 209.7
```
For more `tpcc` options, use `cockroach workload run tpcc --help`. For details about other workloads built into the `cockroach` binary, use `cockroach workload --help`.
1. To monitor the load generator's progress, open the DB Console by pointing a browser to the address in the `admin` field in the standard output of any node on startup.
Since the load generator is pointed at the load balancer, the connections will be evenly distributed across nodes. To verify this, click **Metrics** on the left, select the **SQL** dashboard, and then check the **SQL Connections** graph. You can use the **Graph** menu to filter the graph for specific nodes.
## Step 10. Monitor the cluster
Despite CockroachDB's various built-in safeguards against failure, it is critical to actively monitor the overall health and performance of a cluster running in production and to create alerting rules that promptly send notifications when there are events that require investigation or intervention.
For details about available monitoring options and the most important events and metrics to alert on, see Monitoring and Alerting.
## Step 11. Scale the cluster
You can start the nodes manually or automate the process using [systemd](https://www.freedesktop.org/wiki/Software/systemd/).
For each additional node you want to add to the cluster, complete the following steps:
1. SSH to the machine where you want the node to run.
2. Install CockroachDB for Linux.
3. Run the `cockroach start` command, passing the new node's address as the `--advertise-addr` flag and pointing `--join` to the three existing nodes (also include `--locality` if you set it earlier).
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--advertise-addr=
For each additional node you want to add to the cluster, complete the following steps:
1. SSH to the machine where you want the node to run. Ensure you are logged in as the `root` user.
2. Install CockroachDB for Linux.
3. Create the Cockroach directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
mkdir /var/lib/cockroach
```
4. Create a Unix user named `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
useradd cockroach
```
5. Move the `certs` directory to the `cockroach` directory.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
mv certs /var/lib/cockroach/
```
6. Change the ownership of the `cockroach` directory to the user `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
chown -R cockroach /var/lib/cockroach
```
7. Download the [sample configuration template](https://raw.githubusercontent.com/cockroachlabs/docs/master/_includes/v26.3/prod-deployment/securecockroachdb.service):
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl -o securecockroachdb.service https://raw.githubusercontent.com/cockroachlabs/docs/master/_includes/v26.3/prod-deployment/securecockroachdb.service
```
Alternatively, you can create the file yourself and copy the script into it:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
[Unit]
Description=Cockroach Database cluster node
Requires=network.target
[Service]
Type=notify
WorkingDirectory=/var/lib/cockroach
ExecStart=/usr/local/bin/cockroach start --certs-dir=certs --advertise-addr=\
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
9. Repeat these steps for each additional node that you want in your cluster.
## Step 12. Use the database
Now that your deployment is working, you can:
1. Implement your data model.
2. Create users and grant them privileges.
3. Connect your application. Be sure to connect your application to the load balancer, not to a CockroachDB node.
4. Take backups of your data.
You may also want to adjust the way the cluster replicates data. For example, by default, a multi-node cluster replicates all data 3 times; you can change this replication factor or create additional rules for replicating individual databases and tables differently. For more information, see Replication Controls.
When running a cluster of 5 nodes or more, it's safest to increase the replication factor for important internal data to 5, even if you do not do so for user data. For the cluster as a whole to remain available, the ranges for this internal data must always retain a majority of their replicas.
## See also
* Production Checklist
* Manual Deployment
* Orchestrated Deployment
* Monitoring and Alerting
* Performance Tuning
* Local Deployment
# Deploy CockroachDB On-Premises
Source: https://docs.cockroachlabs.com/docs/v26.3/deploy-cockroachdb-on-premises
This tutorial shows you how to manually deploy a secure multi-node CockroachDB cluster on multiple machines, using [HAProxy](http://www.haproxy.org/) load balancers to distribute client traffic.
If you are only testing CockroachDB, or you are not concerned with protecting network communication with TLS encryption, you can use an insecure cluster instead. Select **Insecure** above for instructions.
To try CockroachDB Cloud instead of running CockroachDB yourself, refer to the Cloud Quickstart.
## Before you begin
### Requirements
* You must have CockroachDB installed locally. This is necessary for generating and managing your deployment's certificates.
* You must have [SSH access]() to each machine. This is necessary for distributing and starting CockroachDB binaries.
* Your network configuration must allow TCP communication on the following ports:
* `26257` for intra-cluster and client-cluster communication
* `8080` to expose your DB Console
* Carefully review the Production Checklist, including supported hardware and software, and the recommended Topology Patterns.
* Do not run multiple node processes on the same VM or machine. This defeats CockroachDB's replication and causes the system to be a single point of failure. Instead, start each node on a separate VM or machine.
* To start a node with multiple disks or SSDs, provide a separate `--store` flag for each disk when starting the `cockroach` process on the node. For more details about stores, see Start a Node.
If you start a node with multiple `--store` flags, it is not possible to scale back down to only using a single store on the node. Instead, you must decommission the node and start a new node with the updated `--store`.
* When starting each node, use the `--locality` flag to describe the node's location, for example, `--locality=region=west,zone=us-west-1`. The key-value pairs should be ordered from most to least inclusive, and the keys and order of key-value pairs must be the same on all nodes.
* When deploying in a single availability zone:
* To be able to tolerate the failure of any 1 node, use at least 3 nodes with the `default` 3-way replication factor. In this case, if 1 node fails, each range retains 2 of its 3 replicas, a majority.
* To be able to tolerate 2 simultaneous node failures, use at least 5 nodes and increase the `default` replication factor for user data to 5. The replication factor for important internal data is 5 by default, so no adjustments are needed for internal data. In this case, if 2 nodes fail at the same time, each range retains 3 of its 5 replicas, a majority.
* When deploying across multiple availability zones:
* To be able to tolerate the failure of 1 entire AZ in a region, use at least 3 AZs per region and set `--locality` on each node to spread data evenly across regions and AZs. In this case, if 1 AZ goes offline, the 2 remaining AZs retain a majority of replicas.
* To ensure that ranges are split evenly across nodes, use the same number of nodes in each AZ. This is to avoid overloading any nodes with excessive resource consumption.
* When deploying across multiple regions:
* To be able to tolerate the failure of 1 entire region, use at least 3 regions.
### Recommendations
* Decide how you want to access your DB Console:
| Access Level | Description |
| - | - |
| Partially open | Set a firewall rule to allow only specific IP addresses to communicate on port `8080`. |
| Completely open | Set a firewall rule to allow all IP addresses to communicate on port `8080`. |
| Completely closed | Set a firewall rule to disallow all communication on port `8080`. In this case, a machine with SSH access to a node could use an SSH tunnel to access the DB Console. |
## Step 1. Synchronize clocks
CockroachDB requires moderate levels of clock synchronization to preserve data consistency. For this reason, when a node detects that its clock is out of sync with at least half of the other nodes in the cluster by 80% of the maximum offset allowed (500ms by default), it spontaneously shuts down. This avoids the risk of consistency anomalies, but it's best to prevent clocks from drifting too far in the first place by running clock synchronization software on each node.
[`ntpd`](http://doc.ntp.org/) should keep offsets in the single-digit milliseconds, so that software is featured here, but other methods of clock synchronization are suitable as well.
1. SSH to the first machine.
2. Disable `timesyncd`, which tends to be active by default on some Linux distributions:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo timedatectl set-ntp no
```
Verify that `timesyncd` is off:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ timedatectl
```
Look for `Network time on: no` or `NTP enabled: no` in the output.
3. Install the `ntp` package:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo apt-get install ntp
```
4. Stop the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp stop
```
5. Sync the machine's clock with Google's NTP service:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpd -b time.google.com
```
To make this change permanent, in the `/etc/ntp.conf` file, remove or comment out any lines starting with `server` or `pool` and add the following lines:
```
server time1.google.com iburst
server time2.google.com iburst
server time3.google.com iburst
server time4.google.com iburst
```
Restart the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp start
```
We recommend Google's NTP service because it handles ["smearing" the leap second](https://developers.google.com/time/smear). If you use a different NTP service that doesn't smear the leap second, be sure to configure client-side smearing in the same way on each machine. See the Production Checklist for details.
1. Verify that the machine is using a Google NTP server:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpq -p
```
The active NTP server will be marked with an asterisk.
2. Repeat these steps for each machine where a CockroachDB node will run.
Compute Engine instances are preconfigured to use [NTP](http://www.ntp.org/), which should keep offsets in the single-digit milliseconds. However, Google can’t predict how external NTP services, such as `pool.ntp.org`, will handle the leap second. Therefore, you should:
* [Configure each GCE instance to use Google's internal NTP service](https://cloud.google.com/compute/docs/instances/configure-ntp#configure_ntp_for_your_instances).
* If you plan to run a hybrid cluster across GCE and other cloud providers or environments, note that all of the nodes must be synced to the same time source, or to different sources that implement leap second smearing in the same way. See the Production Checklist for details.
Amazon provides the [Amazon Time Sync Service](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html), which uses a fleet of satellite-connected and atomic reference clocks in each AWS Region to deliver accurate current time readings. The service also smears the leap second.
* [Configure each AWS instance to use the internal Amazon Time Sync Service](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html#configure-amazon-time-service).
* Per the above instructions, ensure that `etc/chrony.conf` on the instance contains the line `server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4` and that other `server` or `pool` lines are commented out.
* To verify that Amazon Time Sync Service is being used, run `chronyc sources -v` and check for a line containing `* 169.254.169.123`. The `*` denotes the preferred time server.
* If you plan to run a hybrid cluster across GCE and other cloud providers or environments, note that all of the nodes must be synced to the same time source, or to different sources that implement leap second smearing in the same way. See the Production Checklist for details.
[`ntpd`](http://doc.ntp.org/) should keep offsets in the single-digit milliseconds, so that software is featured here. However, to run `ntpd` properly on Azure VMs, it's necessary to first unbind the Time Synchronization device used by the Hyper-V technology running Azure VMs; this device aims to synchronize time between the VM and its host operating system but has been known to cause problems.
1. SSH to the first machine.
2. Find the ID of the Hyper-V Time Synchronization device:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ curl -O https://raw.githubusercontent.com/torvalds/linux/master/tools/hv/lsvmbus
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ python lsvmbus -vv | grep -w "Time Synchronization" -A 3
```
```
VMBUS ID 12: Class_ID = {9527e630-d0ae-497b-adce-e80ab0175caf} - [Time Synchronization]
Device_ID = {2dd1ce17-079e-403c-b352-a1921ee207ee}
Sysfs path: /sys/bus/vmbus/devices/2dd1ce17-079e-403c-b352-a1921ee207ee
Rel_ID=12, target_cpu=0
```
3. Unbind the device, using the `Device_ID` from the previous command's output:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ echo | sudo tee /sys/bus/vmbus/drivers/hv_utils/unbind
```
4. Install the `ntp` package:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo apt-get install ntp
```
5. Stop the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp stop
```
6. Sync the machine's clock with Google's NTP service:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpd -b time.google.com
```
To make this change permanent, in the `/etc/ntp.conf` file, remove or comment out any lines starting with `server` or `pool` and add the following lines:
```
server time1.google.com iburst
server time2.google.com iburst
server time3.google.com iburst
server time4.google.com iburst
```
Restart the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp start
```
We recommend Google's NTP service because it handles ["smearing" the leap second](https://developers.google.com/time/smear). If you use a different NTP service that doesn't smear the leap second, be sure to configure client-side smearing in the same way on each machine. See the Production Checklist for details.
7. Verify that the machine is using a Google NTP server:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpq -p
```
The active NTP server will be marked with an asterisk.
8. Repeat these steps for each machine where a CockroachDB node will run.
## Step 2. Generate certificates
You can use `cockroach cert` commands or `openssl` commands to generate security certificates. This section features the `cockroach cert` commands.
Locally, you'll need to create the following certificates and keys:
* A certificate authority (CA) key pair (`ca.crt` and `ca.key`).
* A node key pair for each node, issued to its IP addresses and any common names the machine uses, as well as to the IP addresses and common names for machines running load balancers.
* A client key pair for the `root` user. You'll use this to run a sample workload against the cluster as well as some `cockroach` client commands from your local machine.
1. Install CockroachDB on your local machine, if you haven't already.
2. Create two directories:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir certs
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir my-safe-directory
```
* `certs`: You'll generate your CA certificate and all node and client certificates and keys in this directory and then upload some of the files to your nodes.
* `my-safe-directory`: You'll generate your CA key in this directory and then reference the key when generating node and client certificates. After that, you'll keep the key safe and secret; you will not upload it to your nodes.
3. Create the CA certificate and key:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-ca \
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
4. Create the certificate and key for the first node, issued to all common names you might use to refer to the node as well as to the load balancer instances:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-node \
\
\
\
\
localhost \
127.0.0.1 \
\
\
\
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
5. Upload the CA certificate and node certificate and key to the first node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ gcloud compute ssh \
--project \
--command "mkdir certs"
```
`gcloud compute ssh` associates your public SSH key with the GCP project and is only needed when connecting to the first node. See the [GCP docs](https://cloud.google.com/sdk/gcloud/reference/compute/ssh) for more details.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ scp certs/ca.crt \
certs/node.crt \
certs/node.key \
@@@@@
This is necessary because the certificates and keys for additional nodes will also be named `node.crt` and `node.key`. As an alternative to deleting these files, you can run the next `cockroach cert create-node` commands with the `--overwrite` flag.
1. Create the certificate and key for the second node, issued to all common names you might use to refer to the node as well as to the load balancer instances:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-node \
\
\
\
\
localhost \
127.0.0.1 \
\
\
\
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
2. Upload the CA certificate and node certificate and key to the second node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ ssh @@@@@@`cockroach` client commands from your local machine. If you might also want to run `cockroach` client commands directly on a node (e.g., for local debugging), you'll need to copy the `root` user's certificate and key to that node as well.
On accessing the DB Console in a later step, your browser will consider the CockroachDB-created certificate invalid and you’ll need to click through a warning message to get to the UI. You can avoid this issue by using a certificate issued by a public CA.
## Step 3. Start nodes
You can start the nodes manually or automate the process using [systemd](https://www.freedesktop.org/wiki/Software/systemd/).
For each initial node of your cluster, complete the following steps:
After completing these steps, nodes will not yet be live. They will complete the startup process and join together to form a cluster as soon as the cluster is initialized in the next step.
1. Visit Releases and download the full binary of CockroachDB to the node.
2. On the node, follow the instructions to install CockroachDB.
3. Run the `cockroach start` command:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--advertise-addr=
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
| `--cache`
`--max-sql-memory` | Increases the node's cache size to 25% of available system memory to improve read performance. The capacity for in-memory SQL processing defaults to 25% of system memory but can be raised, if necessary, to increase the number of simultaneous client connections allowed by the node as well as the node's capacity for in-memory processing of rows when using `ORDER BY`, `GROUP BY`, `DISTINCT`, joins, and window functions. For more details, see Cache and SQL Memory Size. |
| `--background` | Starts the node in the background so you gain control of the terminal to issue more commands. |
When deploying across multiple datacenters, or when there is otherwise high latency between nodes, it is recommended to set `--locality` as well. For more details, see Locality.
For other flags not explicitly set, the command uses default values. For example, the node stores data in `--store=cockroach-data` and binds DB Console HTTP requests to `--http-addr=Start a Node.
Repeat these steps for each additional node that you want in your cluster.
For each initial node of your cluster, complete the following steps:
After completing these steps, nodes will not yet be live. They will complete the startup process and join together to form a cluster as soon as the cluster is initialized in the next step.
1. SSH to the machine where you want the node to run. Ensure you are logged in as the `root` user.
2. Install CockroachDB for Linux.
3. Create the Cockroach directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir /var/lib/cockroach
```
4. Create a Unix user named `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ useradd cockroach
```
5. Move the `certs` directory to the `cockroach` directory.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mv certs /var/lib/cockroach/
```
6. Change the ownership of the `cockroach` directory to the user `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ chown -R cockroach /var/lib/cockroach
```
7. Download the [sample configuration template](https://raw.githubusercontent.com/cockroachlabs/docs/main/src/current/_includes/v26.3/prod-deployment/securecockroachdb.service) and save the file in the `/etc/systemd/system/` directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl -o securecockroachdb.service https://raw.githubusercontent.com/cockroachlabs/docs/main/src/current/_includes/v23.2/prod-deployment/securecockroachdb.service
```
Alternatively, you can create the file yourself and copy the script into it:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
[Unit]
Description=Cockroach Database cluster node
Requires=network.target
[Service]
Type=notify
WorkingDirectory=/var/lib/cockroach
ExecStart=/usr/local/bin/cockroach start --certs-dir=certs --advertise-addr=\
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
When deploying across multiple datacenters, or when there is otherwise high latency between nodes, it is recommended to set `--locality` as well. For more details, see Locality.
For other flags not explicitly set, the command uses default values. For example, the node stores data in `--store=cockroach-data` and binds DB Console HTTP requests to `--http-addr=localhost:8080`. To set these options manually, see Start a Node.
9. Start the CockroachDB cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
systemctl start securecockroachdb
```
10. Configure `systemd` to start CockroachDB automatically after a reboot:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
systemctl enable securecockroachdb
```
11. Repeat these steps for each additional node that you want in your cluster.
`systemd` handles node restarts in case of node failure. To stop a node without `systemd` restarting it, run `systemctl stop securecockroachdb`
## Step 4. Initialize the cluster
On your local machine, run the `cockroach init` command to complete the node startup process and have them join together as a cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach init --certs-dir=certs --host=standard output, such as the CockroachDB version, the URL for the DB Console, and the SQL URL for clients.
## Step 5. Test the cluster
CockroachDB replicates and distributes data behind-the-scenes and uses a [Gossip protocol](https://wikipedia.org/wiki/Gossip_protocol) to enable each node to locate data across the cluster. Once a cluster is live, any node can be used as a SQL gateway.
When using a load balancer, you should issue commands directly to the load balancer, which then routes traffic to the nodes.
Use the built-in SQL client locally as follows:
1. On your local machine, launch the built-in SQL client, with the `--host` flag set to the address of the load balancer:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --certs-dir=certs --host= CREATE DATABASE securenodetest;
```
3. View the cluster's databases, which will include `securenodetest`:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SHOW DATABASES;
```
```
+--------------------+
| Database |
+--------------------+
| crdb_internal |
| information_schema |
| securenodetest |
| pg_catalog |
| system |
+--------------------+
(5 rows)
```
4. Use `\q` to exit the SQL shell.
## Step 6. Set up load balancing
Each CockroachDB node is an equally suitable SQL gateway to your cluster, but to ensure client performance and reliability, it's important to use load balancing:
* **Performance:** Load balancers spread client traffic across nodes. This prevents any one node from being overwhelmed by requests and improves overall cluster performance (queries per second).
* **Reliability:** Load balancers decouple client health from the health of a single CockroachDB node. In cases where a node fails, the load balancer redirects client traffic to available nodes.
[HAProxy](http://www.haproxy.org/) is one of the most popular open-source TCP load balancers, and CockroachDB includes a built-in command for generating a configuration file that is preset to work with your running cluster, so we feature that tool here.
1. On your local machine, run the `cockroach gen haproxy` command with the `--host` flag set to the address of any node and security flags pointing to the CA cert and the client cert and key:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroach gen haproxy \
--certs-dir=certs \
--host=:26257 check port 8080
server cockroach2 :26257 check port 8080
server cockroach3 :26257 check port 8080
```
The file is preset with the minimal [configurations](http://cbonte.github.io/haproxy-dconv/1.7/configuration.html) needed to work with your running cluster:
| Field | Description |
| - | - |
| `timeout connect`
`timeout client`
`timeout server` | Timeout values that should be suitable for most deployments. |
| `bind` | The port that HAProxy listens on. This is the port clients will connect to and thus needs to be allowed by your network configuration.
This tutorial assumes HAProxy is running on a separate machine from CockroachDB nodes. If you run HAProxy on the same machine as a node (not recommended), you'll need to change this port, as `26257` is likely already being used by the CockroachDB node. |
| `balance` | The balancing algorithm. This is set to `roundrobin` to ensure that connections get rotated amongst nodes (connection 1 on node 1, connection 2 on node 2, etc.). Check the [HAProxy Configuration Manual](http://cbonte.github.io/haproxy-dconv/1.7/configuration.html#4-balance) for details about this and other balancing algorithms. |
| `option httpchk` | The HTTP endpoint that HAProxy uses to check node health. `/health?ready=1` ensures that HAProxy doesn't direct traffic to nodes that are live but not ready to receive requests. |
| `server` | For each included node, this field specifies the address the node advertises to other nodes in the cluster, i.e., the addressed pass in the `--advertise-addr` flag on node startup. Make sure hostnames are resolvable and IP addresses are routable from HAProxy. |
For full details on these and other configuration settings, see the [HAProxy Configuration Manual](http://cbonte.github.io/haproxy-dconv/1.7/configuration.html).
1. Upload the `haproxy.cfg` file to the machine where you want to run HAProxy:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
scp haproxy.cfg @built-in workloads for simulating client traffic. This step features CockroachDB's version of the [TPC-C](http://www.tpc.org/tpcc/) workload.
Be sure that you have configured your network to allow traffic from the application to the load balancer. In this case, you will run the sample workload on one of your machines. The traffic source should therefore be the **internal (private)** IP address of that machine.
1. SSH to the machine where you want to run the sample TPC-C workload.
This should be a machine that is not running a CockroachDB node, and it should already have a `certs` directory containing `ca.crt`, `client.root.crt`, and `client.root.key` files.
2. Install CockroachDB for Linux.
3. Use the `cockroach workload` command to load the initial schema and data, pointing it at the IP address of the load balancer:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload init tpcc \
'postgresql://root@:26257/tpcc?sslmode=verify-full&sslrootcert=certs/ca.crt&sslcert=certs/client.root.crt&sslkey=certs/client.root.key'
```
4. Use the `cockroach workload` command to run the workload for 10 minutes:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload run tpcc \
--duration=10m \
'postgresql://root@:26257/tpcc?sslmode=verify-full&sslrootcert=certs/ca.crt&sslcert=certs/client.root.crt&sslkey=certs/client.root.key'
```
You'll see per-operation statistics print to standard output every second:
```
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
1s 0 1443.4 1494.8 4.7 9.4 27.3 67.1 transfer
2s 0 1686.5 1590.9 4.7 8.1 15.2 28.3 transfer
3s 0 1735.7 1639.0 4.7 7.3 11.5 28.3 transfer
4s 0 1542.6 1614.9 5.0 8.9 12.1 21.0 transfer
5s 0 1695.9 1631.1 4.7 7.3 11.5 22.0 transfer
6s 0 1569.2 1620.8 5.0 8.4 11.5 15.7 transfer
7s 0 1614.6 1619.9 4.7 8.1 12.1 16.8 transfer
8s 0 1344.4 1585.6 5.8 10.0 15.2 31.5 transfer
9s 0 1351.9 1559.5 5.8 10.0 16.8 54.5 transfer
10s 0 1514.8 1555.0 5.2 8.1 12.1 16.8 transfer
...
```
After the specified duration (10 minutes in this case), the workload will stop and you'll see totals printed to standard output:
```
_elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__result
600.0s 0 823902 1373.2 5.8 5.5 10.0 15.2 209.7
```
For more `tpcc` options, use `cockroach workload run tpcc --help`. For details about other workloads built into the `cockroach` binary, use `cockroach workload --help`.
1. To monitor the load generator's progress, open the DB Console by pointing a browser to the address in the `admin` field in the standard output of any node on startup.
Since the load generator is pointed at the load balancer, the connections will be evenly distributed across nodes. To verify this, click **Metrics** on the left, select the **SQL** dashboard, and then check the **SQL Connections** graph. You can use the **Graph** menu to filter the graph for specific nodes.
## Step 8. Monitor the cluster
Despite CockroachDB's various built-in safeguards against failure, it is critical to actively monitor the overall health and performance of a cluster running in production and to create alerting rules that promptly send notifications when there are events that require investigation or intervention.
For details about available monitoring options and the most important events and metrics to alert on, see Monitoring and Alerting.
## Step 9. Scale the cluster
You can start the nodes manually or automate the process using [systemd](https://www.freedesktop.org/wiki/Software/systemd/).
For each additional node you want to add to the cluster, complete the following steps:
1. SSH to the machine where you want the node to run.
2. Install CockroachDB for Linux.
3. Run the `cockroach start` command, passing the new node's address as the `--advertise-addr` flag and pointing `--join` to the three existing nodes (also include `--locality` if you set it earlier).
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--advertise-addr=
For each additional node you want to add to the cluster, complete the following steps:
1. SSH to the machine where you want the node to run. Ensure you are logged in as the `root` user.
2. Install CockroachDB for Linux.
3. Create the Cockroach directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
mkdir /var/lib/cockroach
```
4. Create a Unix user named `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
useradd cockroach
```
5. Move the `certs` directory to the `cockroach` directory.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
mv certs /var/lib/cockroach/
```
6. Change the ownership of the `cockroach` directory to the user `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
chown -R cockroach /var/lib/cockroach
```
7. Download the [sample configuration template](https://raw.githubusercontent.com/cockroachlabs/docs/master/_includes/v26.3/prod-deployment/securecockroachdb.service):
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl -o securecockroachdb.service https://raw.githubusercontent.com/cockroachlabs/docs/master/_includes/v26.3/prod-deployment/securecockroachdb.service
```
Alternatively, you can create the file yourself and copy the script into it:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
[Unit]
Description=Cockroach Database cluster node
Requires=network.target
[Service]
Type=notify
WorkingDirectory=/var/lib/cockroach
ExecStart=/usr/local/bin/cockroach start --certs-dir=certs --advertise-addr=\
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
9. Repeat these steps for each additional node that you want in your cluster.
## Step 10. Use the cluster
Now that your deployment is working, you can:
1. Implement your data model.
2. Create users and grant them privileges.
3. Connect your application. Be sure to connect your application to the load balancer, not to a CockroachDB node.
4. Take backups of your data.
You may also want to adjust the way the cluster replicates data. For example, by default, a multi-node cluster replicates all data 3 times; you can change this replication factor or create additional rules for replicating individual databases and tables differently. For more information, see Replication Controls.
When running a cluster of 5 nodes or more, it's safest to increase the replication factor for important internal data to 5, even if you do not do so for user data. For the cluster as a whole to remain available, the ranges for this internal data must always retain a majority of their replicas.
## See also
* Production Checklist
* Manual Deployment
* Orchestrated Deployment
* Monitoring and Alerting
* Performance Tuning
* Local Deployment
# Deploy CockroachDB with the CockroachDB Operator
Source: https://docs.cockroachlabs.com/docs/v26.3/deploy-cockroachdb-with-cockroachdb-operator
This page describes how to start and stop a secure 3-node CockroachDB cluster in a single [Kubernetes](http://kubernetes.io/) cluster.
## Prerequisites and best practices
### Kubernetes version
To deploy CockroachDB v25.1 or later, Kubernetes 1.30 or higher is required. Cockroach Labs strongly recommends that you use a Kubernetes version that is eligible for [patch support by the Kubernetes project](https://kubernetes.io/releases/).
### Helm version
The CockroachDB Helm chart requires Helm 3.0 or higher. If you attempt to use an incompatible Helm version, an error like the following occurs:
```
Error: UPGRADE FAILED: template: cockroachdb/templates/tests/client.yaml:6:14: executing "cockroachdb/templates/tests/client.yaml" at <.Values.networkPolicy.enabled>: nil pointer evaluating interface {}.enabled
```
There are two Helm charts that must be deployed:
* `operator`: The CockroachDB operator chart to be installed first.
* `cockroachdb`: The CockroachDB application chart to be installed after the operator is ready.
### Network
Service Name Indication (SNI) is an extension to the TLS protocol that allows a client to indicate which hostname it is attempting to connect to at the start of the TCP handshake process. The server can present multiple certificates on the same IP address and TCP port number, and one server can serve multiple secure websites or API services even if they use different certificates.
Due to its order of operations, the PostgreSQL wire protocol's implementation of TLS is incompatible with SNI-based routing in the Kubernetes ingress controller. Instead, use a TCP load balancer for CockroachDB that is not shared with other services.
If you want to secure your cluster to use TLS certificates for all network communications, Helm must be installed with RBAC privileges. Otherwise, you will get an `attempt to grant extra privileges` error.
### Localities
CockroachDB clusters use localities to efficiently distribute replicas. This is especially important for [multi-region deployments](#multi-region-deployments). With the CockroachDB operator, you specify mappings between locality levels and the location on a Kubernetes node where the value for that locality can be found.
In cloud provider deployments (e.g., [GKE](#hosted-gke), [EKS](#hosted-eks), or [AKS](#hosted-aks)), the [`topology.kubernetes.io/region`](https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesioregion) and [`topology.kubernetes.io/zone`](https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone) values on Kubernetes nodes are populated by cloud provider. For further granularity, you can define arbitrary locality labels (e.g., `province`, `datacenter`, `rack`), but these need to be applied individually to the Kubernetes node when initialized so that CockroachDB can understand where the node lives and distribute replicas accordingly.
On bare metal Kubernetes deployments, you must plan a hierarchy of localities that suit your CockroachDB node distribution, then apply these values individually to nodes when they are initialized. Although you can set most of these values arbitrarily, you must set region and zone locations in the reserved `topology.kubernetes.io/region` and `topology.kubernetes.io/zone` namespaces, respectively.
For more information on how locality labels are used by CockroachDB, refer to the `--locality` documentation.
### Architecture
The CockroachDB operator is only supported in environments with an ARM64 or AMD64 architecture.
### Resources
When starting Kubernetes, select machines with at least 4 vCPUs and 16 GiB of memory, and provision at least 2 vCPUs and 8 GiB of memory to CockroachDB per pod. These minimum settings are used by default in this deployment guide, and are appropriate for testing purposes only. On a production deployment, you should adjust the resource settings for your workload.
### Storage
Kubernetes deployments use external persistent volumes that are often replicated by the provider. CockroachDB replicates data automatically, and this redundant layer of replication can impact performance. Using [local volumes](https://kubernetes.io/docs/concepts/storage/volumes/#local) may improve performance.
### Multi-region deployments
The following deployment requirements and best practices apply to multi-region deployments with the CockroachDB operator:
* Plan an operator deployment in each region. Each operator handles CockroachDB node management in its own region, so you must factor that there are multiple operator deployments into your upgrade and maintenance strategy to ensure availability.
* VPC peering and CoreDNS are needed to resolve services and discovery across regions. Refer to the documentation for your cloud solution to configure your cross-region network accordingly.
* Make sure that region is part of your [locality](#localities) configuration and nodes are tagged consistently with their deployment region. For deployments with a cloud service provider this is handled automatically, but must be applied manually for bare metal deployments.
* A single certificate authority (CA) across all regions is required. One namespace per region is recommended to simplify certificate management.
## Step 1. Start Kubernetes
You can use the hosted [Google Kubernetes Engine (GKE)](#hosted-gke) service, hosted [Amazon Elastic Kubernetes Service (EKS)](#hosted-eks), or [Microsoft Azure Kubernetes Service (AKS)](#hosted-aks) to quickly start Kubernetes.
Cloud providers such as GKE, EKS, and AKS are not required to run CockroachDB on Kubernetes. You can use any cluster hardware with the minimum recommended Kubernetes version and at least 3 pods, each presenting sufficient resources to start a CockroachDB node. However, note that support for other deployments may vary.
You can also evaluate a CockroachDB Kubernetes deployment on a local machine with [minikube](https://minikube.sigs.k8s.io/docs/start/).
### Hosted GKE
1. Complete the **Before You Begin** steps described in the [Google Kubernetes Engine Quickstart](https://cloud.google.com/kubernetes-engine/docs/quickstart) documentation.
This includes installing `gcloud`, which is used to create and delete Kubernetes Engine clusters, and `kubectl`, which is the command-line tool used to manage Kubernetes from your workstation.
The documentation offers the choice of using Google's Cloud Shell product or using a local shell on your machine. Choose to use a local shell if you want to be able to view the DB Console using the steps in this guide.
2. From your local workstation, start the Kubernetes cluster, specifying one of the available [regions](https://cloud.google.com/compute/docs/regions-zones#available) (e.g., `us-east1`).
The process can take a few minutes, so do not move on to the next step until you see a `Creating cluster cockroachdb...done` message and details about your cluster.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
gcloud container clusters create cockroachdb --machine-type n2-standard-4 --region {region-name} --num-nodes 1
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
Creating cluster cockroachdb...done.
```
Since this region can differ from your default `gcloud` region, be sure to include the `--region` flag to run `gcloud` commands against this cluster.
This creates GKE instances and joins them into a single Kubernetes cluster named `cockroachdb`. The `--region` flag specifies a [regional three-zone cluster](https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-regional-cluster), and `--num-nodes` specifies one Kubernetes worker node in each zone.
The `--machine-type` flag tells the node pool to use the [n2-standard-4](https://cloud.google.com/compute/docs/machine-types#standard_machine_types) machine type (4 vCPUs, 16 GB memory), which meets our recommended CPU and memory configuration.
Consider creating another, dedicated node group for the operator pod for system resource availability.
1. Get the email address associated with your Google Cloud account:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
gcloud info | grep Account
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
Account: [your.google.cloud.email@example.org]
```
The preceding command returns your email address in all lowercase. However, in the next step, you must enter the address using the accurate capitalization. For example, if your address is `YourName@example.com`, you must use `YourName@example.com` and not `yourname@example.com`.
2. [Create the RBAC roles](https://cloud.google.com/kubernetes-engine/docs/how-to/role-based-access-control#prerequisites_for_using_role-based_access_control) CockroachDB needs for running on GKE, using the address from the previous step:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kubectl create clusterrolebinding $USER-cluster-admin-binding \
--clusterrole=cluster-admin \
--user={your.google.cloud.email@example.org}
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
clusterrolebinding.rbac.authorization.k8s.io/your.username-cluster-admin-binding created
```
### Hosted EKS
1. Complete the steps described in the [EKS Getting Started](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html) documentation.
This includes installing and configuring the AWS CLI and `eksctl`, which is the command-line tool used to create and delete Kubernetes clusters on EKS, and `kubectl`, which is the command-line tool used to manage Kubernetes from your workstation.
If you are running [EKS-Anywhere](https://aws.amazon.com/eks/eks-anywhere/), CockroachDB requires that you [configure your default storage class](https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/) to auto-provision persistent volumes. Alternatively, you can define a custom storage configuration as required by your install pattern.
2. From your local workstation, start the Kubernetes cluster:
To ensure that all 3 nodes can be placed into a different availability zone, you may want to first [confirm that at least 3 zones are available in the region](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#availability-zones-describe) for your account.
Cluster provisioning usually takes between 10 and 15 minutes. Do not move on to the next step until you see a message like `[✔] EKS cluster "cockroachdb" in "us-east-1" region is ready` and details about your cluster.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
eksctl create cluster \
--name cockroachdb \
--nodegroup-name standard-workers \
--node-type m6i.xlarge \
--nodes 3 \
--nodes-min 1 \
--nodes-max 4 \
--node-ami auto
```
This creates EKS instances and joins them into a single Kubernetes cluster named `cockroachdb`. The `--node-type` flag tells the node pool to use the [m6i.xlarge](https://aws.amazon.com/ec2/instance-types/) instance type (4 vCPUs, 16 GB memory), which meets our recommended CPU and memory configuration.
Consider creating another, dedicated node group for the operator pod for system resource availability.
3. Open the [AWS CloudFormation console](https://console.aws.amazon.com/cloudformation/home) to verify that the stacks `eksctl-cockroachdb-cluster` and `eksctl-cockroachdb-nodegroup-standard-workers` were successfully created. Be sure that your region is selected in the console.
### Hosted AKS
1. Complete the **Before you begin**, **Define environment variables**, and **Create a resource groups** steps described in the [AKS quickstart guide](https://learn.microsoft.com/azure/aks/learn/quick-kubernetes-deploy-cli). This includes setting up the Azure CLI and the `az` tool, which is the command-line tool to create and manage Azure cloud resources.
Set the environment variables as desired for your CockroachDB deployment. For these instructions, set the `MY_AKS_CLUSTER_NAME` variable to `cockroachdb`.
Do not follow the **Create an AKS cluster** steps or following sections of the AKS quickstart guide, as these topics will be described specifically for CockroachDB in this documentation.
2. From your workstation, create the Kubernetes cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
az aks create \
--resource-group $MY_RESOURCE_GROUP_NAME \
--name $MY_AKS_CLUSTER_NAME \
--node-count 3 \
--generate-ssh-keys
```
3. Configure authentication for the CockroachbDB operator. The operator can use [Azure Managed Identities](https://learn.microsoft.com/entra/identity/managed-identities-azure-resources/overview) if they are configured.
If Azure Managed Identities are not configured for the CockroachDB operator, create an application in your Azure tenant and create a secret named `azure-cluster-identity-credentials-secret` that contains `AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, and `AZURE_CLIENT_SECRET` to hold the application credentials. You can use the following example YAML to define this application:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
apiVersion: v1
kind: Secret
metadata:
name: azure-cluster-identity-credentials-secret
type: Opaque
stringData:
azure-credentials: |
azure_tenant_id: {tenant ID}
azure_client_id: {client ID}
azure_client_secret: {client secret}
```
For more information on how to use these variables, refer to the [`Azure.Identity` documentation](https://learn.microsoft.com/dotnet/api/azure.identity.environmentcredential?view=azure-dotnet).
### OpenShift deployments
To deploy to an OpenShift cluster, the OpenShift Container Platform must be version 4.17 or later.
### Bare metal deployments
For bare metal deployments, the specific Kubernetes infrastructure deployment steps should be similar to those described in [Hosted GKE](#hosted-gke) and [Hosted EKS](#hosted-eks).
* You must plan a hierarchy of [locality labels](#localities) that suit your CockroachDB node distribution, then apply these labels individually to nodes when they are initialized. Although you can set most of these values arbitrarily, you must set region and zone locations in the reserved `topology.kubernetes.io/region` and `topology.kubernetes.io/zone` namespaces, respectively.
## Step 2. Start CockroachDB
### Add the CockroachDB Helm repository
1. Add the CockroachDB Helm repository:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
helm repo add cockroachdb-v2 https://charts.cockroachdb.com/v2 --force-update
helm repo update cockroachdb-v2
# List available operator chart versions.
helm search repo cockroachdb-v2/cockroachdb-operator-chart --versions
# List available CockroachDB chart versions and pick the desired patch.
# To include release candidate versions of CockroachDB, append the --devel flag to 'helm search'.
helm search repo cockroachdb-v2/cockroachdb-chart --versions
```
Learn more about CockroachDB Helm chart versioning in the [Helm charts repository](https://github.com/cockroachdb/helm-charts/blob/master/cockroachdb-parent/docs/VERSIONING.md).
CockroachDB production charts are also published as OCI artifacts if preferred:
**Google Artifact Registry**:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
helm pull oci://us-docker.pkg.dev/releases-prod/self-hosted/charts/cockroachdb-operator-chart --version 1.0.0
# Google Artifact Registry carries the same chart versions as the Helm repository.
# Run the following command to list the available {CHART_VERSION} numbers:
helm search repo cockroachdb-v2/cockroachdb-chart --versions
helm pull oci://us-docker.pkg.dev/releases-prod/self-hosted/charts/cockroachdb-chart --version {CHART_VERSION}
```
**DockerHub**:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
helm pull oci://registry-1.docker.io/cockroachdb/cockroachdb-operator-chart --version 1.0.0
# DockerHub carries the same chart versions as the Helm repository.
# Run the following command to list the available {CHART_VERSION} numbers:
helm search repo cockroachdb-v2/cockroachdb-chart --versions
helm pull oci://registry-1.docker.io/cockroachdb/cockroachdb-chart --version {CHART_VERSION}
```
The DockerHub repositories must exist before pulling charts unless DockerHub auto-create is enabled.
2. Set your environment variables. This step is optional but recommended in order to use the example commands and templates described in the following instructions. Note the default Kubernetes namespace of `cockroach-ns`.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export CRDBOPERATOR=crdb-operator
export CRDBCLUSTER=cockroachdb
export NAMESPACE=cockroach-ns
```
3. Install the operator sub-chart:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kubectl create namespace $NAMESPACE
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
helm install $CRDBOPERATOR cockroachdb-v2/cockroachdb-operator-chart --version 1.0.0 -n $NAMESPACE
```
### Initialize the cluster
1. Generate a values file that tells Helm how to configure the Kubernetes cluster with the following command:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
helm show values cockroachdb-v2/cockroachdb-chart --version {CHART_VERSION} > values.yaml
```
2. Open `values.yaml` in a text editor.
3. Modify the `cockroachdb.crdbCluster.regions` section to describe the number of CockroachDB nodes to deploy and what region(s) to deploy them in. Replace the default `cloudProvider` with the appropriate value (`gcp`, `aws`, `azure`). For bare metal deployments, you can remove the `cloudProvider` field. The following example initializes three nodes on Google Cloud in the `us-central1` region:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroachdb:
crdbCluster:
regions:
- code: us-central1
nodes: 3
cloudProvider: gcp
namespace: cockroach-ns
```
If you intend to deploy CockroachDB nodes across multiple different regions, follow the additional steps described in [Deploy across multiple regions](#deploy-across-multiple-regions).
1. Modify `cockroachdb.crdbCluster.podTemplate.spec.resources` in the values file with the CPU and memory limits and requests for each node to use. For example, to define default values of 4vCPU and 16GiB of memory:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroachdb:
crdbCluster:
podTemplate:
spec:
resources:
limits:
cpu: 4000m
memory: 16Gi
requests:
cpu: 4000m
memory: 16gi
```
For more information on configuring node resource allocation, refer to Resource management
2. Modify the TLS configuration as desired. For a secure deployment, set `cockroachdb.tls.enabled` in the values file to `true`. By default, the certificates are created automatically by a self-signer utility, which requires no configuration beyond setting a custom certificate duration if desired. This utility uses `cockroach cert` to automatically generate self-signed certificates for the nodes and root client which are stored in a secret. You can see these certificates by running `kubectl get secrets`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kubectl get secrets
```
```
crdb-cockroachdb-ca-secret Opaque 2 23s
crdb-cockroachdb-client-secret kubernetes.io/tls 3 22s
crdb-cockroachdb-node-secret kubernetes.io/tls 3 23s
```
If you are deploying on OpenShift you must also set `cockroachdb.tls.selfSigner.securityContext.enabled` to `false` to mitigate stricter security policies.
If your organization's security policy requires that you use a custom CA certificate or otherwise use your own certificates, read the certificate management documentation.
1. In `cockroachdb.crdbCluster.localityMappings`, provide [locality mappings](#localities) that define locality levels and map them to node labels where the locality information of each Kubernetes node is stored. When CockroachDB is initialized on a node, it processes these values as though they are provided through the `cockroach start --locality` flag.
If `localityMappings` is not configured, by default the CockroachDB operator uses the `region` and `zone` locality labels, mapped implicitly to the [`topology.kubernetes.io/region`](https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesioregion) and [`topology.kubernetes.io/zone`](https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone) node labels.
* In cloud provider deployments, the `topology.kubernetes.io/region` and `topology.kubernetes.io/zone` values on a node are populated by the cloud provider.
* In bare metal deployments, the `topology.kubernetes.io/region` and `topology.kubernetes.io/zone` node label values are not set implicitly by a cloud provider when initializing the node, so you must set them manually or configure custom locality labels.
To add more granular levels of locality to your nodes or use different locality labels, add custom locality levels as values in the `cockroachdb.crdbCluster.localityMappings` list. Any custom `localityMappings` configuration overrides the default `region` and `zone` configuration, so if you append an additional locality level but wish to keep the `region` and `zone` labels you must declare them manually.
The following example uses the existing `region` and `zone` labels and adds an additional `datacenter` locality mapping that is more granular than `zone`. This example declares that the `dc` locality information is stored in the `example.datacenter.locality` node label:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroachdb:
crdbCluster:
localityMappings:
- nodeLabel: "topology.kubernetes.io/region"
localityLabel: "region"
- nodeLabel: "topology.kubernetes.io/zone"
localityLabel: "zone"
- nodeLabel: "example.datacenter.locality"
localityLabel: "dc"
```
The list of `localityMappings` is processed in a top-down hierarchy, where each entry is processed as a lower locality level than the previous locality. In this example, if a Kubernetes node is initialized in the `us-central1` region, `us-central1-c` zone, and `dc2` datacenter, its `cockroach start --locality` flag would be equivalent to the following:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroach start --locality region=us-central1,zone=us-central1-c,dc=dc2
```
Optionally, review the `cockroachdb.crdbCluster.podTemplate.spec.topologySpreadConstraints` configuration and set `topologyKey` to the `nodeLabel` value of a locality level that has distinct values for each node. By default the lowest locality level is `zone`, so the following configuration sets that value as the `topologyKey`:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroachdb:
crdbCluster:
podTemplate:
spec:
topologySpreadConstraints:
# maxSkew defines the degree to which the pods can be unevenly distributed.
- maxSkew: 1
# topologyKey defines the key for topology spread.
topologyKey: topology.kubernetes.io/zone
```
For more information on localities and topology planning, see the topology patterns documentation.
2. Modify other relevant parts of the configuration such as other `topologySpreadConstraints` fields, `service.ports`, and others as needed for your configuration.
3. Run the following command to install the CockroachDB chart using Helm:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
helm install $CRDBCLUSTER cockroachdb-v2/cockroachdb-chart --version {CHART_VERSION} -f values.yaml -n $NAMESPACE
```
You can override the default parameters using the `--set key=value[,key=value]` argument while installing the chart:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
helm install $CRDBCLUSTER cockroachdb-v2/cockroachdb-chart --version {CHART_VERSION} -f values.yaml --set clusterDomain=cluster-test.local -n $NAMESPACE
```
4. Verify that the cluster has initialized and pods are now running:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kubectl get pods -n $NAMESPACE
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
NAME READY STATUS RESTARTS AGE
crdb-operator-655fbf7847-zn9v8 1/1 Running 0 10m
cockroachdb-9swcg 2/2 Running 0 45s
cockroachdb-bn6f7 2/2 Running 0 45s
cockroachdb-nk2dw 2/2 Running 0 45s
```
#### Deploy across multiple regions
The Helm chart supports specifying multiple region definitions in `cockroachdb.crdbCluster.regions` with their respective node counts. Make sure to review the prerequisites and best practices for a [multi-region deployment](#multi-region-deployments).
For each region, modify the `regions` configuration as described in [Initialize the cluster](#initialize-the-cluster) and perform `helm install` against the respective Kubernetes cluster.
The following example shows a configuration across two regions, `us-central1` and `us-east1`, with 3 nodes in each cluster:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroachdb:
clusterDomain: cluster.gke.gcp-us-east1
crdbCluster:
regions:
- code: us-central1
nodes: 3
cloudProvider: gcp
domain: cluster.gke.gcp-us-central1
namespace: cockroach-ns
- code: us-east1
nodes: 3
cloudProvider: gcp
domain: cluster.gke.gcp-us-east1
namespace: cockroach-ns
```
While applying the installation in a given region, do the following:
* Verify that the domain matches `cockroachdb.clusterDomain` in the values file.
* Ensure that `cockroachdb.crdbCluster.regions` captures the information for regions that have already been deployed, including the current region. This allows CockroachDB in the current region to connect to clusters deployed in the existing regions.
## Step 3. Use the built-in SQL client
To use the CockroachDB SQL client, follow these steps to launch a secure pod running the `cockroach` binary.
1. Download the secure client Kubernetes application:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl -O https://raw.githubusercontent.com/cockroachdb/helm-charts/master/examples/client-secure.yaml
```
This client tool logs into CockroachDB as `root` using the root certificates.
1. Edit the yaml file with the following values:
* `spec.serviceAccountName: my-release-cockroachdb`
* `spec.image: cockroachdb/cockroach:`
* `spec.volumes[0].project.sources[0].secret.name: my-release-cockroachdb-client-secret`
2. Launch a pod using this file and keep it running indefinitely:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kubectl create -f client-secure.yaml
```
3. Get a shell into the pod and start the CockroachDB built-in SQL client:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kubectl exec -it cockroachdb-client-secure \
-- ./cockroach sql \
--certs-dir=/cockroach/cockroach-certs \
--host=cockroachdb-public
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
# Server version: CockroachDB CCL v21.1.0 (x86_64-unknown-linux-gnu, built 2021/04/23 13:54:57, go1.13.14) (same version as client)
# Cluster ID: a96791d9-998c-4683-a3d3-edbf425bbf11
#
# Enter \? for a brief introduction.
#
root@cockroachdb-public:26257/defaultdb>
```
This pod will continue running indefinitely, so any time you need to reopen the built-in SQL client or run any other cockroach client commands (e.g., cockroach node), repeat this step using the appropriate cockroach command. If you'd prefer to delete the pod and recreate it when needed, run `kubectl delete pod cockroachdb-client-secure`.
4. Run some basic CockroachDB SQL statements:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
CREATE DATABASE bank;
CREATE TABLE bank.accounts (id INT PRIMARY KEY, balance DECIMAL);
INSERT INTO bank.accounts VALUES (1, 1000.50);
SELECT * FROM bank.accounts;
id | balance
+----+---------+
1 | 1000.50
(1 row)
```
5. Create a user with a password:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
CREATE USER roach WITH PASSWORD 'Q7gc8rEdS';
```
You will need this username and password to access the DB Console later.
6. Exit the SQL shell and pod:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
\q
```
## Step 4. Access the DB Console
To access the cluster's DB Console:
1. On secure clusters, certain pages of the DB Console can only be accessed by `admin` users.
Get a shell into the pod and start the CockroachDB built-in SQL client:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kubectl exec -it cockroachdb-client-secure \
-- ./cockroach sql \
--certs-dir=/cockroach/cockroach-certs \
--host=cockroachdb-public
```
2. Assign `roach` to the `admin` role (you only need to do this once):
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
GRANT admin TO roach;
```
3. Exit the SQL shell and pod:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
\q
```
4. In a new terminal window, port-forward from your local machine to the `cockroachdb-public` service:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kubectl port-forward service/cockroachdb-public 8080
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
Forwarding from 127.0.0.1:8080 -> 8080
```
Run the `port-forward` command on the same machine as the web browser in which you want to view the DB Console. If you have been running these commands from a cloud instance or other non-local shell, you will not be able to view the UI without configuring `kubectl` locally and running the preceding `port-forward` command on your local machine.
5. Go to [`https://localhost:8080`](https://localhost:8080/) and log in with the username and password you created earlier.
If you are using Google Chrome, and get an error about not being able to reach `localhost` because its certificate has been revoked, go to `chrome://flags/#allow-insecure-localhost`, enable "Allow invalid certificates for resources loaded from localhost", and then restart the browser. This degrades security for all sites running on `localhost`, not just CockroachDB's DB Console, so enable the feature only temporarily.
1. In the DB Console, verify that the cluster is running as expected:
1. View the **Node List** to ensure that all nodes successfully joined the cluster.
2. Click the **Databases** tab on the left to verify that `bank` is listed.
## Next steps
Read the following pages for detailed information on cluster scaling, certificate management, resource management, best practices, and other cluster operation details:
* Pod scheduling
* Resource management
* Certificate management
* Cluster scaling
* Cluster monitoring
* Upgrade a cluster
* Override deployment templates
* CockroachDB performance on Kubernetes
## Examples
### Authenticate with `cockroach cert`
The following example uses cockroach cert commands to generate and sign the CockroachDB node and client certificates. To learn more about the supported methods of signing certificates, refer to Authentication.
1. Create two directories:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
mkdir certs my-safe-directory
```
2. Create the CA certificate and key pair:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroach cert create-ca \
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
3. Create a client certificate and key pair for the root user:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroach cert create-client root \
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
4. Upload the client certificate and key to the Kubernetes cluster as a secret, renaming them to the filenames required by the CockroachDB operator:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kubectl create secret generic cockroachdb.client.root \
--from-file=tls.key=certs/client.root.key \
--from-file=tls.crt=certs/client.root.crt \
--from-file=ca.crt=certs/ca.crt
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
secret/cockroachdb.client.root created
```
5. Create the certificate and key pair for your CockroachDB nodes, specifying the namespace you used when deploying the cluster. This example uses the `cockroach-ns` namespace:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroach cert create-node localhost \
127.0.0.1 \
cockroachdb-public \
cockroachdb-public.cockroach-ns \
cockroachdb-public.cockroach-ns.svc.cluster.local \
*.cockroachdb \
*.cockroachdb.cockroach-ns \
*.cockroachdb.cockroach-ns.svc.cluster.local \
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
6. Upload the node certificate and key to the Kubernetes cluster as a secret, renaming them to the filenames required by the CockroachDB operator:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kubectl create secret generic cockroachdb.node \
--from-file=tls.key=certs/node.key \
--from-file=tls.crt=certs/node.crt \
--from-file=ca.crt=certs/ca.crt
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
secret/cockroachdb.node created
```
7. Check that the secrets were created on the cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kubectl get secrets
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
NAME TYPE DATA AGE
cockroachdb.client.root Opaque 3 13s
cockroachdb.node Opaque 3 3s
default-token-6js7b kubernetes.io/service-account-token 3 9h
```
8. Add `cockroachdb.tls.externalCertificates.certificates.nodeSecretName` and `cockroachdb.tls.externalCertificates.certificates.nodeClientSecretName` to the values file used to deploy the cluster:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroachdb:
tls:
enabled: true
externalCertificates:
enabled: true
certificates:
nodeSecretName: cockroachdb.node
nodeClientSecretName: cockroachdb.client.root
```
# Create and Deploy an AWS Lambda Function Built on CockroachDB
Source: https://docs.cockroachlabs.com/docs/v26.3/deploy-lambda-function
This tutorial shows you how to create an [AWS Lambda](https://aws.amazon.com/lambda) function that communicates with a CockroachDB Standard cluster.
## Before you begin
Before starting the tutorial, do the following:
1. Create an [AWS](https://aws.amazon.com/) account and log in, then:
1. Create an [AWS user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users.html) with administrator permissions.
2. Install the [AWS CLI](https://aws.amazon.com/cli/) and log in locally.
## Step 1. Create a CockroachDB Standard cluster
1. Create a CockroachDB Cloud account. If this is your first CockroachDB Cloud organization, it will be credited with \$400 in free trial credits to get you started.
2. On the **Get Started** page, click **Create cluster**.
3. On the **Select a plan** page, select **Standard**.
4. On the **Cloud & Regions** page, select a cloud provider (GCP or AWS).
5. In the **Regions** section, select a region for the cluster. Refer to CockroachDB Cloud Regions for the regions where CockroachDB Standard clusters can be deployed. To create a multi-region cluster, click **Add region** and select additional regions.
6. Click **Next: Capacity**.
7. On the **Capacity** page, keep the **Provisioned capacity** at the default value of 2 vCPUs.
Click **Next: Finalize**.
8. On the **Finalize** page, name your cluster. If an active free trial is listed in the right pane, you will not need to add a payment method, though you will need to do this by the end of the trial to maintain your organization's clusters.
Click **Create cluster**.
Your cluster will be created in a few seconds and the **Create SQL user** dialog will display.
After the cluster is created, the **Connection info** window appears. Click the **Connection string** tab and copy the connection string to a secure location. You will use this connection string to connect to CockroachDB later in the tutorial.
The connection string is pre-populated with your SQL username and password, the cluster name, and other details. Save the password in a secure place such a password manager. You can find the connection string at any time from the CockroachDB Cloud Console. You can reset your SQL password for this cluster from the cluster's **SQL Users** page.
## Step 2. Get the code
In a terminal, clone the [sample code's GitHub repo](https://github.com/cockroachlabs/examples-aws-lambda):
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
git clone https://github.com/cockroachlabs/examples-aws-lambda
```
This repo includes samples for Node.js and Python [Lambda runtimes](https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html). Select either node.js or Python to continue.
The Node.js function code is located in the `examples-aws-lambda/node` directory and has the following structure:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
├── README.md
├── deployment-package.zip ## Lambda deployment package
├── index.js ## Lambda function source code
├── package-lock.json ## Dependencies
└── package.json ## Dependencies
```
The Lambda function uses the [node-postgres](https://node-postgres.com/) modules to connect to your cluster.
The Python function code is located in the `examples-aws-lambda/python` directory and has the following structure:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
├── README.md
├── deployment-package.zip ## Lambda deployment package
├── init_db.py ## Lambda function source code
├── package ## Psycopg dependencies
├── requirements.txt ## List of Python requirements
└── root.crt ## CA cert
```
The Lambda function uses the [Psycopg2](https://www.psycopg.org/) PostgreSQL adapter to connect to your cluster.
## Step 3. (Optional) Create the deployment package
Creating a deployment package to deploy the sample function is optional. The `examples-aws-lambda` repo includes deployment packages that are ready to deploy.
1. In the `node` directory, install the code dependencies:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cd node; npm install
```
2. Compress the project files to a ZIP file in the parent directory for deployment:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
zip -r ../my-deployment-package.zip .
```
3. In the `python` directory, download and install the `psycopg2-binary` Python library:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cd python; python3 -m pip install \
--only-binary :all: \
--platform manylinux1_x86_64 \
--target ./my-package \
-r requirements.txt
```
To run on Amazon Linux distributions, `pscyopg2` dependencies must be compiled for Linux.
1. Compress the project files to a ZIP file in the parent directory for deployment:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
zip -r ../my-deployment-package.zip my-deployment-package ./init_db.py ./root.crt
```
## Step 4. Configure AWS
1. Configure the AWS CLI to authenticate with your AWS account:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
aws configure
```
Follow the prompts to authenticate as a user with administrator privileges. We do not recommend using the `root` user.
2. Create an execution role for the Lambda function and attach the `AWSLambdaBasicExecutionRole` policy to the role. The Lambda function needs this role to run.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
aws iam create-role \
--role-name lambda-ex \
--assume-role-policy-document '{"Version": "2012-10-17","Statement": [{ "Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole"}]}'
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
aws iam attach-role-policy \
--role-name lambda-ex \
--policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
```
## Step 5. Deploy the function to AWS Lambda
1. In the deployment package directory, use the AWS CLI to create a Lambda function:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
aws lambda create-function \
--function-name init-crdb \
--region \
--zip-file fileb://deployment-package.zip \
--handler index.handler \
--runtime nodejs14.x \
--role arn:aws:iam:: \
--zip-file fileb://deployment-package.zip \
--handler init_db.lambda_handler \
--runtime python3.9 \
--role arn:aws:iam::
To connect to a CockroachDB Standard cluster with Psycopg2, you must provide the client with a valid CA certificate. By default, Pscyopg2 searches for the certificate at `~/.postgresql/root.crt` and in the location point to by the environment variable `PGSSLROOTCERT`.
1. Invoke the function:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
aws lambda invoke \
--function-name init-crdb out \
--log-type Tail \
--query 'LogResult' \
--output text | base64 -d
```
```
START RequestId: 12232b98-daac-4a1e-80e9-f2ecaa6497aa Version: $LATEST
2022-02-16T19:24:49.569Z 12232b98-daac-4a1e-80e9-f2ecaa6497aa INFO Initializing table...
2022-02-16T19:24:49.596Z 12232b98-daac-4a1e-80e9-f2ecaa6497aa INFO Hey! You successfully connected to your CockroachDB cluster.
2022-02-16T19:24:49.635Z 12232b98-daac-4a1e-80e9-f2ecaa6497aa INFO Created new account with balance 320.
2022-02-16T19:24:49.655Z 12232b98-daac-4a1e-80e9-f2ecaa6497aa INFO Created new account with balance 593.
2022-02-16T19:24:49.660Z 12232b98-daac-4a1e-80e9-f2ecaa6497aa INFO Created new account with balance 277.
2022-02-16T19:24:49.675Z 12232b98-daac-4a1e-80e9-f2ecaa6497aa INFO Created new account with balance 844.
2022-02-16T19:24:49.680Z 12232b98-daac-4a1e-80e9-f2ecaa6497aa INFO Created new account with balance 257.
2022-02-16T19:24:49.680Z 12232b98-daac-4a1e-80e9-f2ecaa6497aa INFO Database initialized.
END RequestId: 12232b98-daac-4a1e-80e9-f2ecaa6497aa
REPORT RequestId: 12232b98-daac-4a1e-80e9-f2ecaa6497aa Duration: 644.49 ms Billed Duration: 645 ms Memory Size: 128 MB Max Memory Used: 63 MB Init Duration: 198.77 ms
```
## See also
* Build a Simple Django App with CockroachDB
You might also be interested in the following pages:
* Client Connection Parameters
* Connection Pooling
* Data Replication
* CockroachDB Resilience
* Replication & Rebalancing
# Deployment & Operations Skills Taxonomy
Source: https://docs.cockroachlabs.com/docs/v26.3/deployment-operations-skills-taxonomy
This document outlines the foundational skills required to deploy and operate CockroachDB in production environments.
The skills are organized into sections based on the following operational domains:
* [Infrastructure configuration](#infrastructure-configuration)
* [Security](#security)
* [Cluster maintenance](#cluster-maintenance)
* [Troubleshooting](#troubleshooting)
* [Disaster recovery](#disaster-recovery)
Each section includes links to relevant documentation for the listed skills.
Cockroach Labs offers [Professional Services](https://www.cockroachlabs.com/company/professional-services/) that can assist you with getting applications into production faster and more efficiently.
## Infrastructure configuration
This section covers how to ensure that your hardware and network are properly configured to meet the performance and connectivity requirements of CockroachDB.
* Verify vCPU, RAM, storage, and disk IOPS performance
* Configure time synchronization with NTP server
* Validate network connectivity
## Security
This section covers how to secure a CockroachDB deployment, including certificate management, load balancing setup, role-based access control, and data encryption.
* Create and distribute certificates; initialize cluster
* Configure load balancer and direct a workload
* Configure RBAC
* Encryption at rest
## Cluster maintenance
This section covers how to manage the lifecycle of CockroachDB nodes, including adding and removing nodes, handling outages, performing upgrades or downgrades, and modifying cluster settings.
* Shut down a node gracefully
* Handle unplanned node outages
* Add nodes
* Remove nodes
* Add a region
* Remove a region
* Rolling upgrades
* Downgrade a cluster from a patch version
* Downgrade a cluster from a major version
* Change a cluster setting
* Repave a cluster: cluster repaving involves the following individual skills, which are also used during rolling upgrades:
1. Shut down a node gracefully
2. Detach the persistent volume (a.k.a. persistent disk) from the removed node's virtual machine (VM) (this step is optional but recommended)
3. Delete the removed node's VM
4. Start a new VM
5. Reattach the persistent disk to the new VM (necessary if you did step #2)
6. Add a node to the cluster from the new VM
## Troubleshooting
This section contains a list of common issues related to SQL performance, cluster stability, memory usage, load balancing, and changefeed lag.
* SQL response time for specific queries
* SQL throughput degradation across the board
* Cluster instability: Dead/suspect nodes
* Out of memory (OOM) problems
* Imbalanced cluster load
* End of file (EOF) errors
* Changefeed is falling behind
* Gather diagnostic data from a "debug zip" file
* Collect timeseries diagnostic data from a "tsdump" file
## Disaster recovery
This section covers how to set up and manage backup and restore of your cluster to ensure data recovery in case of failures.
* Create AWS IAM access key
* Create S3 bucket for backup data
* Full cluster backup to S3
* Incremental backup to S3
* Cluster restore from AWS S3
## See also
* Production Checklist
* Manual Deployment
* Deploy a Local Cluster from Binary (Secure)
* SQL Performance Best Practices
* Performance Tuning Recipes
* Troubleshoot Self-Hosted Setup
# Developer Basics
Source: https://docs.cockroachlabs.com/docs/v26.3/developer-basics
Before you start using CockroachDB, you should understand the following important mechanics of the database:
* [How transactions work in CockroachDB](#how-transactions-work-in-cockroachdb), including [serializability](#serializability-and-transaction-contention) and [transaction retries](#transaction-retries)
* [How applications interact with CockroachDB](#how-applications-interact-with-cockroachdb)
Note that the sections that follow were written for the purposes of orienting application developers. For more detailed technical documentation on CockroachDB's architecture, see our architecture documentation.
### How transactions work in CockroachDB
CockroachDB is designed to make your data scalable and consistent. All stored records are replicated across a distributed deployment of database instances, and all database transactions committed in CockroachDB satisfy [ACID properties](https://en.wikipedia.org/wiki/ACID).
To guarantee that database operations are [atomic](https://wikipedia.org/wiki/Atomicity_\(database_systems\)) (the "A" of the [ACID properties](https://en.wikipedia.org/wiki/ACID)), CockroachDB executes all statements in the context of an atomic [database transaction](https://wikipedia.org/wiki/Database_transaction). If a transaction succeeds, all data mutations are applied together simultaneously from the perspective of operations outside of the transaction. If any part of a transaction fails, the entire transaction is aborted, and the database is left unchanged.
Managing transactions is an important part of CockroachDB application development. When creating your application's persistence layer, you must understand how CockroachDB interacts with your application framework's transaction management system. You should also have some knowledge of CockroachDB's transaction syntax. We go into more detail about transaction management in Manage Transactions.
#### Serializability and transaction contention
By default, CockroachDB uses `SERIALIZABLE` transaction [isolation](https://wikipedia.org/wiki/Isolation_\(database_systems\)) (the "I" of ACID semantics). If transactions are executed concurrently, the final state of the database will appear as if the transactions were executed serially. `SERIALIZABLE` isolation, the strictest level of isolation, provides the highest level of data consistency and protects against concurrency anomalies.
To guarantee data consistency, CockroachDB locks the data targeted by an open transaction. If a separate transaction attempts to modify data that are locked by an open transaction, the newest transaction will not succeed, as committing it could result in incorrect data. This scenario is called *transaction contention*, and should be avoided when possible. For a more detailed explanation of transaction contention, and tips on how to avoid it, see Transaction contention.
#### Transaction retries
In some cases, transaction contention is unavoidable. If a transaction fails due to contention, CockroachDB will automatically retry the transaction. If the transaction is `SERIALIZABLE` and cannot automatically be retried without allowing serialization anomalies, CockroachDB will return a transaction retry error to the client.
Most official CockroachDB client libraries include a transaction-retrying wrapper function to make writing your persistence layer easier. If your framework's client library does not include a retry wrapper, you will need to write transaction retry logic in your application. We go into more detail about transaction retries later in the guide, in Retry Transactions.
#### Read Committed isolation
CockroachDB can be configured to execute transactions at `READ COMMITTED` instead of `SERIALIZABLE` isolation. `READ COMMITTED` permits some concurrency anomalies in exchange for minimizing transaction aborts and removing the need for client-side retries. Depending on your workload requirements, this may be desirable. For more information, see Read Committed Transactions.
### How applications interact with CockroachDB
CockroachDB supports the PostgreSQL wire protocol, and CockroachDB supports most PostgreSQL syntax and features, making most PostgreSQL client libraries compatible with CockroachDB. Cockroach Labs also maintains official CockroachDB adapters for some of the most popular PostgreSQL drivers and ORMs. Using an official CockroachDB adapter can help simplify transaction management and performance optimization in your application. If an official CockroachDB adapter does not yet exist for your application's data framework, we recommend just using the PostgreSQL client library that works for your application.
A growing number of popular third-party database tools offer full support for CockroachDB. Because CockroachDB uses the PostgreSQL wire protocol and supports PostgreSQL syntax, most PostgreSQL-compatible third-party tools are also compatible with CockroachDB. For a list of libraries and tools that we have tested against CockroachDB, see Third-party Database Tools.
# Disaster Recovery Overview
Source: https://docs.cockroachlabs.com/docs/v26.3/disaster-recovery-overview
Resilient deployments aim for continuity in database operations to protect from data loss and downtime. CockroachDB's built-in Raft replication and fault tolerance provide high availability. However, it is still important to design a *disaster recovery* plan to recover from unforeseen incidents to minimize downtime and data loss.
As you evaluate CockroachDB's disaster recovery features, consider your organization's requirements for the amount of tolerable data loss and the acceptable length of time to recover.
* **Recovery Point Objective (RPO)**: The maximum amount of data loss (measured by time) that an organization can tolerate.
* **Recovery Time Objective (RTO)**: The maximum length of time it should take to restore normal operations following an outage.
When you use backups, RPO and RTO can be visualized as follows:
![Simulating RPO and RTO. With RPO representing the tolerable data loss, and RTO representing the tolerable time to recovery.]()
For an overview of resiliency features in CockroachDB, refer to Data Resilience.
## Choose a disaster recovery strategy
CockroachDB is designed to recover automatically; however, building backups or physical cluster replication into your disaster recovery planning protects against unforeseen incidents.
| | Point-in-time backup & restore | Physical cluster replication (asynchronous) |
| - | - | - |
| **RPO** | >=5 minutes | 10s of seconds |
| **RTO** | Minutes to hours, depending on data size and number of nodes | Seconds to minutes, depending on cluster size, and time of failover |
| **Write latency** | No impact | No impact |
| **Recovery** | Manual restore | Manual failover |
| **Fault tolerance** | Not applicable | Zero RPO node, availability zone within a cluster, region failures with loss up to RPO in a two-region (or two-datacenter) setup |
| **Minimum regions to achieve fault tolerance** | 1 | 2 |
## See also
* Set Up Physical Cluster Replication
* Physical Cluster Replication Technical Overview
* Backup Architecture
* Backup and Restore Overview
* Managed Backups
# CockroachDB Docs MCP Server
Source: https://docs.cockroachlabs.com/docs/v26.3/docs-mcp-integration
The CockroachDB Docs MCP Server ([Model Context Protocol](https://modelcontextprotocol.io/introduction)) enables AI assistants like Claude Desktop, Cursor, and VS Code to access CockroachDB documentation directly. Self-hosted customers can also integrate the server into custom agents and automation workflows using an [authenticated endpoint](#authenticated-endpoint-for-automation).
## Setup
Connect your AI assistant to CockroachDB documentation by configuring the MCP server. The following sections describe platform-specific instructions.
### Cursor
1. Create or open the file `.cursor/mcp.json` in your project directory.
2. Add the following configuration:
```json theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
{
"mcpServers": {
"cockroachdb-docs": {
"type": "http",
"url": "https://cockroachdb.mcp.kapa.ai"
}
}
}
```
3. Restart Cursor to apply the configuration.
4. You can now ask questions about CockroachDB documentation directly in Cursor.
### VS Code
Requires VS Code 1.102 or later with GitHub Copilot.
1. Create or open the file `.vscode/mcp.json` in your project directory.
2. Add the following configuration:
```json theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
{
"servers": {
"cockroachdb-docs": {
"type": "http",
"url": "https://cockroachdb.mcp.kapa.ai"
}
}
}
```
3. Reload VS Code window (⌘+R).
4. The CockroachDB documentation is now available through GitHub Copilot in VS Code.
### Claude Code
1. Run the following command in your terminal:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
claude mcp add --transport http cockroachdb-docs https://cockroachdb.mcp.kapa.ai
```
2. Execute `/mcp` in Claude Code and authenticate via browser when prompted.
CockroachDB documentation is now accessible in your Claude Code environment.
### ChatGPT Desktop
1. Open ChatGPT Desktop.
2. Go to **Settings** → **Features** and enable **Developer mode**.
3. Navigate to **Settings** → **MCP Servers**.
4. Click **Add Server** and enter:
* **Name**: `cockroachdb-docs`
* **URL**: `https://cockroachdb.mcp.kapa.ai`
5. CockroachDB documentation is now accessible in ChatGPT Desktop.
### Claude Desktop
1. Open Claude Desktop configuration file:
* **macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`
* **Windows**: `%APPDATA%\Claude\claude_desktop_config.json`
* **Linux**: `~/.config/Claude/claude_desktop_config.json`
2. Add the CockroachDB Docs MCP server to your configuration:
```json theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
{
"mcpServers": {
"cockroachdb-docs": {
"type": "npx",
"args": ["mcp-remote", "https://cockroachdb.mcp.kapa.ai"]
}
}
}
```
3. Save the file and restart Claude Desktop.
4. You can now query CockroachDB documentation directly in Claude Desktop.
### Other MCP clients
For other MCP-compatible clients, use the following server configuration:
**Server URL**: `https://cockroachdb.mcp.kapa.ai`
**Generic configuration format**:
```json theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
{
"mcpServers": {
"cockroachdb-docs": {
"url": "https://cockroachdb.mcp.kapa.ai"
}
}
}
```
Consult your client's documentation for specific configuration instructions.
## Authenticated endpoint for automation
For self-hosted CockroachDB customers who want to integrate the Docs MCP Server into custom agents or automation workflows, an authenticated endpoint is available. This endpoint requires your CockroachDB license key.
The authenticated endpoint is currently available to self-hosted CockroachDB customers.
### Configuration
Use the following endpoint with your CockroachDB license key:
**Server URL**: `https://docs-mcp.cockroachlabs.com/mcp`
**Generic configuration format**:
```json theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
{
"mcpServers": {
"cockroachdb-docs": {
"url": "https://docs-mcp.cockroachlabs.com/mcp",
"headers": {
"Authorization": "Bearer {YOUR_LICENSE_KEY}"
}
}
}
}
```
Replace `\{YOUR_LICENSE_KEY\}` with your CockroachDB license key.
### Authentication errors (401)
If you receive a `401 Unauthorized` error, check the following:
1. Verify your license key is correct. Refer to the licensing FAQ for details on obtaining your key.
2. Check that the `Authorization` header is formatted as `Bearer \{YOUR_LICENSE_KEY\}` (note the space after `Bearer`).
3. Ensure your license has not expired.
4. If you recently received your license, it may take up to 24 hours to be activated in our system.
## Capabilities
Once connected, your AI assistant can answer questions about CockroachDB directly from the official documentation. Example queries include:
### Multi-region configuration
* "How do I set up a multi-region CockroachDB cluster?"
* "What are the best practices for configuring region survival goals?"
* "Show me how to implement follow-the-workload patterns"
### Performance optimization
* "What are the recommended techniques for optimizing query performance?"
* "How do I identify and resolve hot spots in my cluster?"
* "Explain the best practices for index design in CockroachDB"
### Security features
* "How do I configure TLS/SSL encryption for my cluster?"
* "What are the steps to implement role-based access control?"
* "Show me how to set up audit logging in CockroachDB"
### SQL syntax and features
* "What's the syntax for creating a changefeed?"
* "How do I use window functions in CockroachDB?"
* "Explain the differences between JSONB operators in CockroachDB"
### Troubleshooting
* "How do I diagnose and fix connection refused errors?"
* "What steps should I take when encountering transaction retry errors?"
* "How do I investigate and resolve node liveness issues?"
### Operations and maintenance
* "What's the recommended backup strategy for production clusters?"
* "How do I perform a rolling upgrade of my cluster?"
* "Show me the best practices for monitoring CockroachDB"
## Troubleshooting
### Connection issues
If your AI assistant cannot connect to the CockroachDB documentation:
1. Verify your internet connection.
2. Check that the MCP server URL is correctly configured.
3. Restart your AI assistant application.
4. Ensure you have the latest version of your AI assistant.
### No results
If queries return no results:
1. Try rephrasing your question to be more specific.
2. Include "CockroachDB" in your query for better context.
3. Check the [CockroachDB documentation](/docs) directly to verify the information exists.
### Configuration not working
If the configuration doesn't seem to take effect:
1. Ensure the configuration file is saved in the correct location.
2. Check for JSON syntax errors in your configuration.
3. Fully quit and restart your application (not just reload).
4. For Claude Desktop: Verify that `npx` is available in your system `PATH`.
## Feedback and support
For issues or feedback about the CockroachDB Docs MCP Server, join the [CockroachDB Community Slack](https://www.cockroachlabs.com/join-community).
## See also
* [Model Context Protocol](https://modelcontextprotocol.io/)
# Install a FIPS-ready CockroachDB Runtime
Source: https://docs.cockroachlabs.com/docs/v26.3/fips
**What "FIPS support" means**
When CockroachDB documentation refers to "FIPS support" or "FIPS-ready" deployments, this means CockroachDB can be configured to use FIPS 140-3-approved cryptographic algorithms and operate in accordance with a FIPS 140-3 cryptographic module's Security Policy. It does not mean that CockroachDB itself is FIPS 140-3 validated.
CockroachDB uses the frozen Go Cryptographic Module v1.0.0, which is FIPS 140-3 validated under CMVP Certificate #5247.
## Overview of FIPS-ready CockroachDB
[Federal Information Processing Standards (FIPS) 140-3](https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.140-3.pdf) is a U.S. government standard that specifies security requirements for cryptographic modules. FIPS 140-3 provides measurable security guidelines for protecting sensitive but unclassified information. The standard is applicable to all federal agencies that use cryptographic-based security systems to protect sensitive information in computer and telecommunication systems. U.S. and Canadian governments, as well as organizations working with them, may be subject to FIPS 140-3 requirements.
The Cryptographic Module Validation Program (CMVP) validates cryptographic modules to FIPS 140-3 and other cryptography-based standards. When a cryptographic module or library has a FIPS 140-3 certificate, it has been tested and formally validated under the CMVP as meeting the requirements for FIPS 140-3.
FIPS-ready CockroachDB binaries and Docker images are available for CockroachDB v23.1.0 and later. FIPS-ready CockroachDB runtimes run on Intel 64-bit Linux systems.
Starting with v26.1, FIPS-ready CockroachDB binaries are built using Go's native FIPS 140-3 support. The cryptographic operations are performed by Go's built-in cryptographic modules, which are independent of the host operating system's libraries. This represents a significant architectural change from previous versions (v25.4 and earlier), which used Red Hat's golang-fips toolchain with OpenSSL.
**Migration from FIPS 140-2 to FIPS 140-3**
Previous versions of CockroachDB (v25.4 and earlier) supported FIPS 140-2. Starting with v26.1, CockroachDB uses a cryptographic module designed for FIPS 140-3 support. Starting with v26.2, CockroachDB uses `GOFIPS140=v1.0.0`, the frozen module version that is FIPS 140-3 validated under CMVP Certificate #5247.
FIPS 140-2 will transition to historical status on September 22, 2026, per [NIST's FIPS 140-3 Transition Effort](https://csrc.nist.gov/Projects/fips-140-3-transition-effort).
For details about cryptographic algorithms and key lengths used by CockroachDB, refer to [Details About Cryptographic Algorithms](#details-about-cryptographic-algorithms).
### FIPS-ready features
When you use a FIPS-ready CockroachDB runtime, Cockroach Labs has verified that cryptographic operations in the following contexts meet the requirements of FIPS 140-3:
* Encryption At Rest
* Encrypted Backups
* Change Data Capture to Kafka over TLS
* Certificate-based Node-to-Node and Client-to-Node Authentication
* SASL SCRAM-SHA-256 Password Authentication
* SQL Cryptographic Built-in Functions
When running a FIPS-ready runtime, Cockroach Labs recommends that you avoid using cryptographic operations that are not supported by FIPS 140-3. For example, generating an MD5 hash is not compatible with FIPS 140-3, because MD5 is not a FIPS-validated algorithm. Use algorithms and functions that do not comply with the standard at your own risk.
This page shows how to install and configure a FIPS-ready CockroachDB runtime using Go's native FIPS 140-3 support.
### Performance considerations
When comparing performance of the same workload in a FIPS-ready CockroachDB runtime to a standard CockroachDB runtime, some performance difference may be observed. The amount of performance impact depends upon the workload, cluster configuration, query load, and other factors.
### Upgrading to a FIPS-ready CockroachDB runtime
Upgrading an existing CockroachDB cluster's binaries in-place to be FIPS-ready is not supported.
## Operating system requirements
FIPS-ready CockroachDB uses Go's native cryptographic module, which is independent of the host operating system's libraries. The FIPS-ready binary can run on any Intel 64-bit Linux system.
The FIPS-ready CockroachDB Docker images are based on [Red Hat's Universal Base Image 10](https://catalog.redhat.com/software/containers/ubi10/ubi-minimal/). To [use the FIPS-ready CockroachDB Docker image](#use-the-fips-ready-cockroachdb-docker-image), skip directly to that section of this page.
### Extend Red Hat's Universal Base Image 10 Docker image
If you do not want to use the FIPS-ready CockroachDB Docker image directly, you can create a custom Docker image based on [Red Hat's Universal Base Image 10](https://catalog.redhat.com/software/containers/ubi10/ubi-minimal/):
* The FIPS-ready binary includes the FIPS 140-3 Go Cryptographic Module and does not require additional system libraries to be installed.
## Download FIPS-ready runtimes
FIPS-ready runtimes are not yet available for CockroachDB v26.3. Download links will be added when v26.3.0 becomes available for self-hosted deployments.
## Install the FIPS-ready CockroachDB runtime
When FIPS-ready CockroachDB v26.3 binaries become available, install them in the same way as standard binaries. Refer to Install CockroachDB on Linux.
**Upgrade from v25.4 or v26.1 FIPS to v26.2 or later**
CockroachDB v26.1 and later use Go's native FIPS cryptographic module, a significant architectural change from the OpenSSL-based approach used in v25.4 and earlier. FIPS support was Preview in v26.1, so deployments using FIPS on v25.4 or v26.1 should upgrade to v26.2 or a later version.
Upgrading an existing CockroachDB cluster's binary in-place from non-FIPS to FIPS is not supported. Instead, you can restore your cluster to a new FIPS-ready cluster.
### Verify that CockroachDB is FIPS-ready
To verify that the CockroachDB binary is FIPS-ready, use the `cockroach version` command and check for the `FIPS enabled` field:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroach version | grep -i fips
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
FIPS enabled: true
```
This indicates that CockroachDB was built with Go's native FIPS 140-3 support.
**Change from previous versions:** In v25.4 and earlier, FIPS-ready binaries showed `fips` appended to the Go version (e.g., `go1.19.5fips`). Starting with v26.1, FIPS status is indicated by the `FIPS enabled: true` field.
## Use the FIPS-ready CockroachDB Docker image
FIPS-ready CockroachDB Docker images are not yet available for CockroachDB v26.3. Image tags and usage instructions will be added when v26.3.0 becomes available for self-hosted deployments.
## Details about cryptographic algorithms
This section provides more information about the cryptographic algorithms and key lengths used by FIPS-ready CockroachDB.
### Authentication
#### Inter-node and node identity
**Algorithm**: [TLS 1.3 (RFC 8446)](https://www.rfc-editor.org/rfc/rfc8446).
Refer to Using Digital Certificates with CockroachDB.
#### Client identity
##### Password authentication
**Algorithm**: `bcrypt` or `scram-sha-256`.
Refer to SASL/SCRAM-SHA-256 Secure Password-based Authentication.
##### Client certificates
**Algorithm**: [TLS 1.3 (RFC 8446)](https://www.rfc-editor.org/rfc/rfc8446).
Refer to Using Digital Certificates with CockroachDB.
##### GSSAPI / Kerberos
Not supported for FIPS-ready deployments.
##### SASL / SCRAM password authentication
**Algorithm**: `scram-sha-256`.
Refer to SASL/SCRAM-SHA-256 Secure Password-based Authentication.
##### JSON Web Tokens (JWTs)
**Algorithms**: Specified by the `server.jwt_authentication.jwks` cluster setting.
Refer to Cluster Single Sign-on (SSO) using a JSON web token (JWT).
##### DB Console authentication via OIDC
**Algorithm**: Specified by the identity provider (IdP) as part of the OIDC handshake process.
Refer to Single Sign-on (SSO) for DB Console.
##### HTTP API access via login tokens
**Algorithm**: `sha256` ([RFC 6234](https://datatracker.ietf.org/doc/html/rfc6234)).
### Encryption
#### In flight
**Algorithm**: [TLS 1.3 (RFC 8446)](https://www.rfc-editor.org/rfc/rfc8446).
**Key sizes**: Depends upon the cipher suite in use:
* TLS 1.2:
* `tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256`
* `tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384`
* `tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256`
* `tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384`
* If the `COCKROACH_TLS_ENABLE_OLD_CIPHER_SUITES` environment variable is set:
* `tls.TLS_RSA_WITH_AES_128_GCM_SHA256`
* `tls.TLS_RSA_WITH_AES_256_GCM_SHA384`
* TLS 1.3:
* `TLS_AES_128_GCM_SHA256`
* `TLS_AES_256_GCM_SHA384`
#### At rest
##### Customer-managed backups
##### AWS
Default encryption provided by the [AWS Encryption SDK](https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/introduction.html).
##### GCP
Default encryption provided by [Google Cloud](https://cloud.google.com/docs/security/encryption/default-encryption).
##### Data Encryption at Rest (EAR)
**Algorithm**: Advanced Encryption Standard (AES) encryption, in counter (CTR) mode.
**Key sizes**: The store key is specified by the user, and can be of length 16, 24, or 32 bytes (corresponding to AES-128, AES-192, or AES-256). Data keys are the same length as the store key.
## See also
* Install CockroachDB
* Releases
# Build a GraphQL Application Using Hasura
Source: https://docs.cockroachlabs.com/docs/v26.3/hasura-getting-started
This tutorial shows you how to create a GraphQL application using [Hasura Cloud](https://cloud.hasura.io) and CockroachDB Standard.
The Hasura GraphQL Engine creates [GraphQL](https://graphql.org/) schemas and resolvers based on the tables and views in your CockroachDB cluster, allowing you to submit GraphQL queries to access and manipulate your data.
Hasura GraphQL Engine v2.16 and later supports CockroachDB v22.2 and later.
## Overview
This tutorial will show you how to configure a Hasura project with a CockroachDB data source, allow connections to your CockroachDB cluster from Hasura Cloud, and create a table that is then used with Hasura GraphQL Engine to retrieve and modify data.
## Before you begin
Before you start this tutorial, you need:
* An existing CockroachDB Cloud cluster, running CockroachDB v22.2 or later.
* A [Hasura Cloud account](https://hasura.io/docs/latest/getting-started/getting-started-cloud/).
## Create a new project in Hasura Cloud
1. In the Hasura Cloud console, select **Projects**, then click **New Project**.
2. In the **Create Project** panel select the cloud infrastructure provider and region.
The cloud infrastructure provider and region should match your cluster. For example, if you created a CockroachDB Standard cluster in GCP's `us-east1` region, choose a GCP region closest to `us-east1`.
3. Click **Create Free Project**.
### Add environment variables to your project
You will store your cluster root certificate and connection string in your Hasura project's environment variables. Storing this connection information in environment variables is considered a best practice.
Create a `SSL_ROOT_CERT` environment variable for your cluster's CA cert.
1. Select **Env vars** in your project settings, and click **New Env Var**.
2. Under **Key** type `SSL_ROOT_CERT`, then press **Enter**.
3. Copy the contents of your cluster's CA certificate file you downloaded earlier.
For example, on Mac you can copy the contents of the CA certificate in a terminal using `pbcopy`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
pbcopy < $HOME/.postgresql/root.crt
```
4. Paste the contents of the CA certificate file under **Value**.
5. Click **Add**.
6. Select **Env vars** in your project settings, and click **New Env Var**.
7. Under **Key** type `SSL_ROOT_CERT`, then press **Enter**.
8. Copy the contents of your CockroachDB Standard cluster's CA certificate file you downloaded earlier.
For example, on Mac you can copy the contents of the CA certificate in a terminal using `pbcopy`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
pbcopy < $HOME/Library/CockroachCloud/certs/fake-puppy-ca.crt
```
9. Paste the contents of the CA certificate file under **Value**.
![Adding the SSL_ROOT_CERT environment variable to the Hasura project]()
10. Click **Add**.
Create a `CRDB_URL` environment variable to store the connection string.
1. Click **New Env Var**.
2. Under **Key** type `CRDB_URL`, then press **Enter**.
3. Paste the connection string for your cluster you copied earlier. Add your SQL user password if necessary to the connection string.
4. Remove the `sslmode` and `sslrootcert` query parameters from the end of the connection string. You will add the SSL settings later.
For example, if your connection string in the CockroachDB Cloud console is:
```
postgresql://maxroach:NotAGoodPassword@fake-puppy-595.g95.cockroachlabs.cloud:26257/defaultdb?sslmode=verify-full
```
You would modify the connection string to:
```
postgresql://maxroach:NotAGoodPassword@fake-puppy-595.g95.cockroachlabs.cloud:26257/defaultdb
```
5. Click **Add**.
### Create a Data Source in Hasura Cloud
1. In the overview page in Hasura Cloud, select **Projects**, find the project you created earlier, and click **Launch Console**.
2. Click **Data** in the console header to open the **Data Manager**.
3. Click **Connect Database**.
4. In the **Connect Existing Database** tab configure the connection to your cluster.
1. Under **Database Display Name** type a name for your connection. Your cluster name is a good option. For example, `fake-puppy`.
2. Set **Data Source Driver** to **CockroachDB**.
3. Select **Environment Variable**, then enter `CRDB_URL` in the **Environment Variable** input box.
![Adding the connection string to the Hasura Data Manager]()
5. Expand **Connection Settings**, then expand **SSL Certificates Settings** and set **SSL Mode** to `verify-full`.
6. Enter `SSL_ROOT_CERT` under **SSL Root Certificate**, the environment variable you configured earlier.
7. Click **Connect Database**.
## Add the Hasura Cloud network to your cluster allowlist
Your CockroachDB Standard cluster needs to be configured to allow incoming client connections from Hasura Cloud.
1. In the Hasura Cloud overview page select **Projects**, then click the **Config** icon for your project.
![Click the project Config icon]()
2. In the **General** section, copy the IP address listed under **Hasura Cloud IP**.
3. In the CockroachDB Cloud console select your cluster, click **Networking**, then **Add Network**.
4. In the **Add Network** dialog, set **Network name** to `Hasura Cloud`.
5. Under **Network**, select **New Network**. Paste the IP address of your Hasura Cloud instance you copied earlier.
6. Check **CockroachDB Client to access the databases**, then click **Apply**.
## Create a table in your CockroachDB cluster
1. Connect to your cluster using the SQL client.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroach sql --url "{connection string}"
```
Where is the connection string for your cluster you copied earlier.
2. In the SQL client, create an `accounts` table.
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
CREATE TABLE IF NOT EXISTS accounts (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
balance INT8
);
```
3. Insert some data into the `accounts` table.
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
UPSERT INTO accounts (balance) VALUES (1000), (250);
```
This will create two accounts, one with $1000 and another with $250.
## Track the `accounts` table in the Hasura Cloud Data Manager
1. In the Hasura Cloud Data Manager click `public` under your connection name.
2. Click the **Track All** button.
3. Click **Yes** when prompted to expose the table to the GraphQL API.
### Create a GraphQL query in the Hasura Cloud API Explorer
In the Hasura Cloud Explorer, you will see the `accounts` table. You can expand `accounts` to construct a GraphQL query against the `accounts` table.
For example, to create a query that finds all accounts with balances greater than or equal to \$500:
1. Enter `GetLargeAccounts` as the query name in the Explorer.
2. Expand `accounts` in the Explorer.
3. Check `balance` and `id`, then expand `where`, then `balance`.
4. Check `_gte` to create a "greater than or equal to" where clause for `balance`, then enter `500`.
This creates the following GraphQL query:
```graphql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
query GetLargeAccounts {
accounts(where: {balance: {_gte: "500"}}) {
balance
id
}
}
```
Click the **Execute Query** button to run the `GetLargeAccounts` GraphQL query. The returned data should look like the following:
```json theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
{
"data": {
"accounts": [
{
"balance": 1000,
"id": "2ed41abf-e831-42df-aff4-f88a6f2b0afc"
}
]
}
}
```
![Executing the GraphQL query in the Hasura Cloud API Explorer]()
## Next steps
Now that you have connected your cluster to Hasura GraphQL Engine, you can use the Explorer to create GraphQL queries to access and modify data in your cluster's tables. You can then use these queries in your application. For example, you can create [Hasura Actions](https://hasura.io/docs/latest/actions/index/?utm_source=cockroachdb) that use these queries. See the [Hasura documentation](https://hasura.io/docs/latest/index/?utm_source=cockroachdb) for detailed information on integrating and deploying Hasura in your applications.
# CockroachDB Docs
Source: https://docs.cockroachlabs.com/docs/v26.3/index
CockroachDB is the SQL database for building global, scalable cloud services that survive disasters.
![icon browser]()
Start a Cloud
cluster
![MCP server]()
Use the Docs
MCP server
Get Started
- Install CockroachDB
- Start a Local Cluster
- Connect to the Database
- Learn CockroachDB SQL
- Explore Capabilities
CockroachDB and AI
- CockroachDB and AI Overview
- Vector Indexes
- Agent Skills
- Docs MCP Server
Deploy
- Production Checklist
- CockroachDB Cloud Deployment
- Kubernetes Overview
- Multi-Region Capabilities
- Cluster Maintenance
Migrate
- Migration Overview
- Migrate from PostgreSQL
- Migrate from MySQL
- Migrate from Oracle
- MOLT Fetch
Security
- Security Overview
- Authentication
- Authorization
- Encryption
Reference
- Architecture
- SQL
- CLI
- DB Console
- Third-Party Tools
Learning Resources
Cockroach University
Free online learning platform covering distributed databases, cloud-native applications, general purpose SQL databases & much more!
Start free course ![icon arrow right]()