--- layout: api page_title: /sys/replication - HTTP API description: >- The '/sys/replication' endpoint focuses on managing general operations in Vault Enterprise replication --- # `/sys/replication` @include 'alerts/enterprise-only.mdx' @include 'alerts/restricted-root.mdx' ## Attempt recovery This endpoint attempts recovery if replication is in an adverse state. For example: an error has caused replication to stop syncing. | Method | Path | | :----- | :------------------------- | | `POST` | `/sys/replication/recover` | ### Sample request ```shell-session $ curl \ --header "X-Vault-Token: ..." \ --request POST \ http://127.0.0.1:8200/v1/sys/replication/recover ``` ### Sample response ```json { "warnings": ["..."] } ``` ## Reindex replication This endpoint reindexes the local data storage. This can cause a very long delay depending on the number and size of objects in the data store. **This endpoint requires 'sudo' capability.** | Method | Path | | :----- | :------------------------- | | `POST` | `/sys/replication/reindex` | ### Parameters - `diff` `(bool: false)` – Enables a slower re-indexing which will perform a key level check to diagnose issues. Defaults false. - `force` `(bool: false)` – Forces a complete re-indexing which only scans data available in the storage. Defaults false. - `skip_flush` `(bool: false)` – Skips the tree flushing stage of the reindex process. This setting can be used to reduce the amount of time the tree is locked during a reindex process. If this node is killed before the full tree has been asynchronously flushed the reindex may not have applied fully and a new reindex may need to be done. Defaults false. ### Sample payload ```json {} ``` ### Sample request ```shell-session $ curl \ --header "X-Vault-Token: ..." \ --request POST \ http://127.0.0.1:8200/v1/sys/replication/reindex ``` ### Sample response ```json { "warnings": ["..."] } ``` ## Check status This endpoint print information about the status of replication (mode, sync progress, etc). This is an unauthenticated endpoint. | Method | Path | | :----- | :------------------------ | | `GET` | `/sys/replication/status` | ### Sample request ```shell-session $ curl \ http://127.0.0.1:8200/v1/sys/replication/status ``` ### Sample response The printed status of the replication environment. As an example, for a performance primary and DR primary node, it will look something like: ```json { "data": { "dr": { "cluster_id": "f2c21cb5-523f-617b-20ac-c913d9154ba6", "known_secondaries": ["3"], "last_wal": 291, "merkle_root": "38543b95d44132138003939addbaf94125ec184e", "mode": "primary", "primary_cluster_addr": "", "secondaries": [ { "api_address": "https://127.0.0.1:49264", "cluster_address": "https://127.0.0.1:49267", "connection_status": "connected", "last_heartbeat": "2020-06-10T15:40:47-07:00", "node_id": "3" } ] }, "performance": { "cluster_id": "1598d434-dfec-1f48-f019-3d22a8075bf9", "known_secondaries": ["2"], "last_wal": 291, "merkle_root": "43f40fc775b40cc76cd5d7e289b2e6eaf4ba138c", "mode": "primary", "primary_cluster_addr": "", "secondaries": [ { "api_address": "https://127.0.0.1:49253", "cluster_address": "https://127.0.0.1:49256", "connection_status": "connected", "last_heartbeat": "2020-06-10T15:40:46-07:00", "node_id": "2" } ] } } } ``` Possible values for `connection_status` are `connected` or `disconnected`. `last_heartbeat` is the timestamp of the last time this node exchanged heartbeats with another node. ### Sample response from performance secondary & DR primary The printed status of the replication environment. As an example, for a performance secondary and DR primary node, it will look something like: ```json { "data": { "dr": { "cluster_id": "e4bfa800-002e-7b6d-14c2-617855ece02f", "known_secondaries": ["4"], "last_wal": 455, "merkle_root": "cdcf796619240ce19dd8af30fa700f64c8006e3d", "mode": "primary", "primary_cluster_addr": "", "secondaries": [ { "api_address": "https://127.0.0.1:49277", "cluster_address": "https://127.0.0.1:49281", "connection_status": "connected", "last_heartbeat": "2020-06-10T15:40:46-07:00", "node_id": "4" } ] }, "performance": { "cluster_id": "1598d434-dfec-1f48-f019-3d22a8075bf9", "known_primary_cluster_addrs": ["https://127.0.0.1:8201"], "last_remote_wal": 291, "merkle_root": "43f40fc775b40cc76cd5d7e289b2e6eaf4ba138c", "corrupted_merkle_tree": false, "last_corruption_check_epoch": "1694456090", "mode": "secondary", "primaries": [ { "api_address": "https://127.0.0.1:49244", "cluster_address": "https://127.0.0.1:8201", "connection_status": "connected", "last_heartbeat": "2020-06-10T15:40:46-07:00" } ], "primary_cluster_addr": "https://127.0.0.1:8201", "secondary_id": "2", "state": "stream-wals" } } } ``` ## `/sys/replication/merkle-check` The Merkle check endpoint prints information about the corruption status of the Merkle tree. The corruption information provides insight about whether any current replication issues can be resolved by reindexing the tree. We recommend consulting the Merkle check endpoint **before** reindexing to ensure the process will be useful as reindexing can be time-consuming and lead to downtime. You must provide an authentication token when calling `replication/merkle-check`. | Method | Path | | :------ | :------------------------------ | | `POST` | `/sys/replication/merkle-check` | ### Sample request ```shell-session $ curl \ http://127.0.0.1:8200/v1/sys/replication/merkle-check ``` ### Sample response The response provides information about the health of the Merkle composite tree. The `merkle_corruption_report` stanza provides information about merkle tree corruption. The `corrupted_root` field indicates whether the root of the tree is corrupted. The `corrupted_tree_map` field identifies any corruption in the roots of its subtrees, including replicated and local subtrees. In the event of corruption within a page or subpage of the tree, `corrupted_tree_map` includes the page number along with a list of corrupted subpage numbers. The status details indicate the extent of corruption within the merkle tree. ```json { "request_id": "d4b2ad1a-6e5f-7f9e-edfe-558eb89a40e6", "lease_id": "", "lease_duration": 0, "renewable": false, "data": { "merkle_corruption_report": { "corrupted_root": false, "corrupted_tree_map": { "1": { "corrupted_index_tuples_map": { "5": { "corrupted": false, "subpages": [ 28 ] } }, "corrupted_subtree_root": false, "root_hash": "DyGc6rQTV9XgyNSff3zimhi3FJM=", "tree_type": "replicated" }, "2": { "corrupted_index_tuples_map": null, "corrupted_subtree_root": false, "root_hash": "EXmRTdfYCZTm5i9wLef9RQqyLCw=", "tree_type": "local" } }, "last_corruption_check_epoch": "2023-09-11T11:25:59.44956-07:00" } } } ```