> ## Documentation Index
> Fetch the complete documentation index at: https://cockroachlabs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Admission Control

export const InternalLink = ({version, path = "", children, ...props}) => {
  let detectedVersion = version || "stable";
  if (typeof window !== 'undefined' && !version) {
    const match = window.location.pathname.match(/\/docs\/([^/]+)/);
    if (match) {
      detectedVersion = match[1];
    }
  }
  const normalizedPath = path.startsWith("/") ? path.slice(1) : path;
  return <a href={`/docs/${detectedVersion}/${normalizedPath}`} {...props}>
      {children}
    </a>;
};

CockroachDB supports an admission control system to maintain cluster performance and availability when some nodes experience high load. When admission control is enabled, CockroachDB sorts request and response operations into work queues by priority, giving preference to higher priority operations. Internal operations critical to node health, like <InternalLink path="cluster-setup-troubleshooting#node-liveness-issues">node liveness heartbeats</InternalLink>, are high priority. The admission control system also prioritizes transactions that hold <InternalLink path="crdb-internal#cluster_locks">locks</InternalLink>, to reduce <InternalLink path="performance-best-practices-overview#transaction-contention">contention</InternalLink> and release locks earlier.

<Note>
  Admission control is not available for CockroachDB Standard or CockroachDB Basic clusters.
</Note>

## Use cases for admission control

A well-provisioned CockroachDB cluster may still encounter performance bottlenecks at the node level, as stateful nodes can develop <InternalLink path="performance-best-practices-overview#hot-spots">hot spots</InternalLink> that last until the cluster rebalances itself. When hot spots occur, they should not cause failures or degraded performance for important work.

This is particularly important for CockroachDB Standard and CockroachDB Basic, where one user tenant cluster experiencing high load should not degrade the performance or availability of a different, isolated tenant cluster running on the same host.

Admission control can help if your cluster has degraded performance due to the following types of node overload scenarios:

* The node has more than 32 runnable goroutines per CPU, visible in the **Runnable goroutines per CPU** graph in the <InternalLink path="ui-overload-dashboard#runnable-goroutines-per-cpu">**Overload** dashboard</InternalLink>.
* The node has a high number of files in level 0 of the Pebble LSM tree, visible in the **LSM L0 Health** graph in the <InternalLink path="ui-overload-dashboard#lsm-l0-health">**Overload** dashboard</InternalLink>.
* The node has high CPU usage, visible in the **CPU percent** graph in the <InternalLink path="ui-overload-dashboard#cpu-percent">**Overload** dashboard</InternalLink>.
* The node is experiencing out-of-memory errors, visible in the **Memory Usage** graph in the <InternalLink path="ui-hardware-dashboard#memory-usage">**Hardware** dashboard</InternalLink>. Even though admission control does not explicitly target controlling memory usage, it can reduce memory usage as a side effect of delaying the start of operation execution when the CPU is overloaded.

<Note>
  Admission control is beneficial when overall cluster health is good but some nodes are experiencing overload. If you see these overload scenarios on many nodes in the cluster, that typically means the cluster needs more resources.
</Note>

## Enable and disable admission control

To enable and disable admission control, use the following <InternalLink path="cluster-settings">cluster settings</InternalLink>:

* `admission.kv.enabled` for work performed by the <InternalLink path="architecture/distribution-layer">KV layer</InternalLink>.
* `admission.sql_kv_response.enabled` for work performed in the SQL layer when receiving <InternalLink path="architecture/distribution-layer">KV responses</InternalLink>.
* `admission.sql_sql_response.enabled` for work performed in the SQL layer when receiving <InternalLink path="architecture/sql-layer#distsql">DistSQL responses</InternalLink>.

When you enable admission control Cockroach Labs recommends that you enable it for **all layers**.

Admission control is enabled  by default for all layers.

## Work queues and ordering

When admission control is enabled, request and response operations are sorted into work queues where the operations are organized by priority and transaction start time.

Higher priority operations are processed first. The criteria for determining higher and lower priority operations is different at each processing layer, and is determined by the CPU and storage I/O of the operation. Write operations in the <InternalLink path="architecture/storage-layer">KV storage layer</InternalLink> in particular are often the cause of performance bottlenecks, and admission control prevents <InternalLink path="architecture/storage-layer#pebble">the Pebble storage engine</InternalLink> from experiencing high <InternalLink path="architecture/storage-layer">read amplification</InternalLink>. Critical cluster operations like node heartbeats are processed as high priority, as are transactions that hold <InternalLink path="crdb-internal#cluster_locks">locks</InternalLink> in order to avoid <InternalLink path="performance-recipes#transaction-contention">contention</InternalLink> and release locks earlier.

The transaction start time is used within the priority queue and gives preference to operations with earlier transaction start times. For example, within the high priority queue operations with an earlier transaction start time are processed first.

### Set quality of service level for a session

In an overload scenario where CockroachDB cannot service all requests, you can identify which requests should be prioritized. This is often referred to as *quality of service* (QoS). Admission control queues work throughout the system. To set the quality of service level on the admission control queues on behalf of SQL requests submitted in a session, use the `default_transaction_quality_of_service` <InternalLink path="set-vars">session variable</InternalLink>. The valid values are `critical`, `background`, and `regular`. Admission control must be enabled for this setting to have an effect.

To increase the priority of subsequent SQL requests, run:

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
SET default_transaction_quality_of_service=critical;
```

To decrease the priority of subsequent SQL requests, run:

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
SET default_transaction_quality_of_service=background;
```

To reset the priority to the default session setting (in between background and critical), run:

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
SET default_transaction_quality_of_service=regular;
```

### Set quality of service level for a transaction

To set the quality of service level for a single <InternalLink path="transactions">transaction</InternalLink>, set the <InternalLink path="set-vars">`default_transaction_quality_of_service` session variable</InternalLink> for just that transaction using the <InternalLink path="set-vars#set-a-variable-for-the-duration-of-a-single-transaction">`SET LOCAL`</InternalLink> statement inside a <InternalLink path="begin-transaction">`BEGIN`</InternalLink> ... <InternalLink path="commit-transaction">`COMMIT`</InternalLink> block as shown below. The valid values are `critical`, `background`, and `regular`.

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
BEGIN;
SET LOCAL default_transaction_quality_of_service = 'regular'; -- Edit to desired level
-- Statements to run in this transaction go here
COMMIT;
```

<a id="best-practices" />

## Limitations

Admission control works on the level of each node, not at the cluster level. The admission control system queues requests until the operations are processed or the request exceeds the timeout value (for example by using <InternalLink path="set-vars#supported-variables">`SET statement_timeout`</InternalLink>). If you specify aggressive timeout values, the system may operate correctly but have low throughput as the operations exceed the timeout value while only completing part of the work. There is no mechanism for preemptively rejecting requests when the work queues are long.

Organizing operations by priority can mean that higher priority operations consume all the available resources while lower priority operations remain in the queue until the operation times out.

## Observe admission control performance

The <InternalLink path="ui-overload-dashboard">DB Console Overload dashboard</InternalLink> shows metrics related to the performance of the admission control system.

## See also

To control the maximum number of non-superuser (<InternalLink path="security-reference/authorization">`root`</InternalLink> user or other <InternalLink path="security-reference/authorization#admin-role">`admin` role</InternalLink>) connections a <InternalLink path="architecture/sql-layer">gateway node</InternalLink> can have open at one time, use the `server.max_connections_per_gateway` <InternalLink path="cluster-settings">cluster setting</InternalLink>.  If a new non-superuser connection would exceed this limit, the error message `"sorry, too many clients already"` is returned, along with error code `53300`.  This may be useful in addition to your admission control settings.