Benchmarks on etcd

Storage Memory Usage Benchmark

Mon, 01 Jan 0001 00:00:00 +0000

Two components of etcd storage consume physical memory. The etcd process allocates an in-memory index to speed key lookup. The process’s page cache, managed by the operating system, stores recently-accessed data from disk for quick re-use.

The in-memory index holds all the keys in a B-tree data structure, along with pointers to the on-disk data (the values). Each key in the B-tree may contain multiple pointers, pointing to different versions of its values. The theoretical memory consumption of the in-memory index can hence be approximated with the formula:

Watch Memory Usage Benchmark

Mon, 01 Jan 0001 00:00:00 +0000

Note

The watch features are under active development, and their memory usage may change as that development progresses. We do not expect it to significantly increase beyond the figures stated below.

A primary goal of etcd is supporting a very large number of watchers doing a massively large amount of watching. etcd aims to support O(10k) clients, O(100K) watch streams (O(10) streams per client) and O(10M) total watchings (O(100) watching per stream). The memory consumed by each individual watching accounts for the largest portion of etcd’s overall usage, and is therefore the focus of current and future optimizations.

Benchmarking etcd v3

Mon, 01 Jan 0001 00:00:00 +0000

Physical machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs
etcd version 2.2.0

etcd Cluster

1 etcd member running in v3 demo mode

Testing

Use etcd v3 benchmark tool.

Performance

reading one single key

key size in bytes	number of clients	read QPS	90th Percentile Latency (ms)
256	1	2716	0.4
256	64	16623	6.1
256	256	16622	21.7

The performance is nearly the same as the one with empty server handler.

Benchmarking etcd v2.2.0-rc-memory

Mon, 01 Jan 0001 00:00:00 +0000

Physical machine

GCE n1-standard-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
7.5 GB memory
2x CPUs

etcd

etcd Version: 2.2.0-rc.0+git
Git SHA: 103cb5c
Go Version: go1.5
Go OS/Arch: linux/amd64

Testing

Start 3-member etcd cluster, each of which uses 2 cores.

The length of key name is always 64 bytes, which is a reasonable length of average key bytes.

Benchmarking etcd v2.2.0-rc

Mon, 01 Jan 0001 00:00:00 +0000

Physical machine

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs

etcd Cluster

3 etcd 2.2.0-rc members, each runs on a single machine.

Detailed versions:

etcd Version: 2.2.0-alpha.1+git
Git SHA: 59a5a7e
Go Version: go1.4.2
Go OS/Arch: linux/amd64

Also, we use 3 etcd 2.1.0 alpha-stage members to form cluster to get base performance. etcd’s commit head is at c7146bd5, which is the same as the one that we use in etcd 2.1 benchmark.

Benchmarking etcd v2.2.0

Mon, 01 Jan 0001 00:00:00 +0000

Physical Machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted as etcd data directory
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs

etcd Cluster

3 etcd 2.2.0 members, each runs on a single machine.

Detailed versions:

etcd Version: 2.2.0
Git SHA: e4561dd
Go Version: go1.5
Go OS/Arch: linux/amd64

Testing

Bootstrap another machine, outside of the etcd cluster, and run the hey HTTP benchmark tool with a connection reuse patch to send requests to each etcd cluster member. See the benchmark instructions for the patch and the steps to reproduce our procedures.

Benchmarking etcd v2.1.0

Mon, 01 Jan 0001 00:00:00 +0000

Physical machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs
etcd version 2.1.0 alpha

etcd Cluster

3 etcd members, each runs on a single machine

Testing

Bootstrap another machine and use the hey HTTP benchmark tool to send requests to each etcd member. Check the benchmark hacking guide for detailed instructions.