TiDB architecture: TiKV storage plane

TiDB is an open-source, MySQL-compatible distributed SQL database: the TiDB process handles SQL and transactions, PD places data and issues timestamps, and TiKV stores rows with Multi-Raft, MVCC, and RocksDB. This post covers architecture (planes and division of labor, Regions, TiKV store) then implementation of codecs, client-go, and primaryKey 2PC—illustrated with a small orders table. Grounded in git/tidb and github.com/tikv/client-go/v2 (TiKV server raftstore/RocksDB at architecture level; in-tree unistore for MVCC behavior where the tikv/tikv tree is absent).

Related: MySQL InnoDB, MySQL DML / locking.

2. Architecture

2.1 Cluster planes and division of labor

Clients speak only to TiDB. Durable rows live in TiKV. PD places data and issues timestamps. TiDB does not open RocksDB; it talks to PD and TiKV over RPC. See the figure at the top of this post.

TiDB server is the compute plane: it accepts the MySQL protocol, parses and plans SQL, encodes rows into keys (tablecodec), holds a per-session membuffer, and drives multi-Region transactions with twoPhaseCommitter / client-go. Nodes are interchangeable behind a load balancer; they own sessions and a RegionCache, not durable row storage.

PD is the control plane: it stores Region metadata, serves TSO (startTS / commitTS), answers Region locate requests, and schedules split / merge / transfer-leader from TiKV heartbeats. It never serves row values.

TiKV is the data plane: each store hosts Region peers (leader or follower), replicates with Raft, applies MVCC, and persists in local RocksDB. Client KV RPCs go to the Region leader; followers hold Raft copies and do not take writes for that Region.

Example. Schema used in later sections (table_id = 100, unique index idx_user = index_id = 1). Keys are logical; on the wire they are tablecodec bytes.

CREATE TABLE orders (
  id BIGINT PRIMARY KEY,
  user_id BIGINT NOT NULL,
  amount DECIMAL(10,2) NOT NULL,
  UNIQUE KEY idx_user (user_id)
);

INSERT INTO orders (id, user_id, amount) VALUES
  (1, 10, 19.90),
  (2, 20,  5.00);

Initial commit at commitTS = 400 (startTS = 390).

SELECT id, amount FROM orders WHERE id = 1;

A write follows the same edges: TiDB buffers mutations and runs Prewrite/Commit; PD supplies TSO and leaders; TiKV leaders apply Lock / Write / Default. Region layout and the TiKV process stack are in §2.2–§2.3; primaryKey 2PC detail is in §3.

2.2 Keyspace and Regions

TiKV is one big sorted map. PD cuts it into Regions:

TiDB encodes SQL rows into that map (tablecodec):

A table’s rows sort together under t{tableID}, so splits and scans can follow table boundaries. The client caches Region→leader in RegionCache and refreshes on epoch / not-leader errors.

Example split for orders (table_id = 100):

SELECT * FROM orders WHERE id = 1;              -- one locate → R1 leader
SELECT * FROM orders WHERE id >= 1 AND id <= 2; -- R1 then R2
SELECT id FROM orders WHERE user_id = 20;       -- R3 index, then R2 row

(Point lookup id = 1 result; range and index queries follow the Region comments above.)

2.3 Inside one TiKV store

TiDB is not a coordinator on a shared RocksDB. Each TiKV process owns Multi-Raft, MVCC apply, and local engines.

Stack (top → bottom)

1. gRPC — kvrpcpb (Get, Prewrite, Commit, Coprocessor, …)
2. Scheduler / latches — conflict control before propose
3. raftstore — many Region peers (leader or follower) on this machine
4. Apply + MVCC — Lock / Write / Default after Raft majority
5. kvdb + raftdb — two RocksDB instances (data CFs vs Raft logs)

kvdb column families (production)

One RocksDB batch is atomic on one store. Keys on other Regions/stores need the distributed txn protocol in §3 (primaryKey 2PC). Raft makes each Region’s replicas agree.

A write through that stack:

UPDATE orders SET amount = 21.00 WHERE id = 1;
-- Query OK, 1 row affected

Path in one line: TiDB → gRPC Prewrite/Commit → R1 leader → latch → Raft → apply MVCC into kvdb. Wrong peer returns not leader. primaryKey atomicity and lock resolve details are in §3.

3. Implementation

3.1 TiDB storage façade

pkg/kv: Storage, Transaction, Snapshot.
TiKVDriver.OpenWithOptions builds PD client + client-go KVStore.
NewTiKVTxn wraps tikv.KVTxn.

SELECT amount FROM orders WHERE id = 1;  -- → Get(EncodeRowKeyWithHandle(100, 1))

3.2 Encoding rows (tablecodec)

tablePrefix     = []byte{'t'}
recordPrefixSep = []byte("_r")
indexPrefixSep  = []byte("_i")

func EncodeRowKey(tableID int64, encodedHandle []byte) kv.Key { /* t + tableID + _r + handle */ }

SELECT id, user_id FROM orders ORDER BY id;

3.3 RegionCache (client-go)

KVStore holds PD client, oracle, RegionCache, lock resolver.

func (c *RegionCache) LocateKey(bo *retry.Backoffer, key []byte) (*KeyLocation, error)

SELECT amount FROM orders WHERE id = 2;

3.4 Distributed MVCC and primaryKey 2PC

TiKV has no central txn table. Each Region stores MVCC locally. For multi-key writes, the client sets one mutated key as primaryKey (twoPhaseCommitter.primaryKey / Lock.Primary). That key’s Write is the durable commit bit. Regions are not typed primary/secondary—only keys in this txn are.

Prewrite:  Lock[key] = { startTS, Primary: primaryKey, ... }
Decision:  Commit(primaryKey) → Write@commitTS
Others:    Commit(secondary) or resolve via Lock.Primary

Flow: Begin (TSO) → choose primaryKey → Prewrite per Region leader → Commit primaryKey first → Commit secondaries → cleanup/resolve.

Single-key example

BEGIN;                                          -- startTS = 410
UPDATE orders SET amount = 21.00 WHERE id = 1; -- primaryKey = t100_r1
COMMIT;                                         -- commitTS = 420

SELECT amount FROM orders WHERE id = 1;  -- startTS' = 430 → 21.00

Reader at startTS' = 415 still sees 19.90.

Multi-Region example (R1 = t100_r1, R2 = t100_r2)

BEGIN;  -- startTS = 440
UPDATE orders SET amount = amount + 1.00 WHERE id IN (1, 2);
-- primaryKey = t100_r1; secondary = t100_r2
COMMIT; -- commitTS = 450

SELECT id, amount FROM orders ORDER BY id;  -- after resolve

Checking primaryKey runs on Lock hit, not on every query.

3.5 Coprocessor

SELECT SUM(amount) FROM orders;

(Amounts 21.00 + 5.00 after the single-row update in §2.3, before the multi-Region +1.00.) TiDB may push a DAG per Region; TiKV computes partial aggregates next to data. Same SQL result whether merged in TiDB or partially pushed down.

3.6 Unistore (in-tree stand-in)

pkg/store/mockstore/unistore/tikv.MVCCStore: LockStore + Badger, Prewrite/Commit aligned with kvrpcpb, no Multi-Raft. Use for lock/resolve behavior tests—not RocksDB CF layout.

-- A: BEGIN; UPDATE orders SET amount = 30 WHERE id = 1;  -- holds Lock
-- B: BEGIN; UPDATE orders SET amount = 40 WHERE id = 1; COMMIT;
-- B waits / conflicts until A commits or rolls back

3.7 Key files

(client-go = module version in git/tidb/go.mod.)

Production raftstore/RocksDB details live in tikv/tikv. SQL tables here are illustrative for orders; TSO and Region IDs are not from a live cluster.