Primary–follower HA with lavik-ctl

Understand the deployment

This guide uses the lavik, lavik-meta and lavik-ctl binaries shipped together in v0.1.0-beta.1. lavik-ctl administers Meta; it does not install a version, launch OS processes, or implement a start/stop service manager. First launch Meta and Data, then use cluster-create to initialize the fresh topology.

The runnable example puts three Meta voters and two Data processes on one Linux host. It demonstrates primary–follower behavior, but a host outage still stops everything. The separate-host deployment section explains the required failure-domain and endpoint changes.

The single-node and HA labs reuse the same ports. Stop the earlier lab before starting the other, and keep its original state directory if you want to return to it.

The lab uses plaintext loopback listeners. --allow-plaintext-admin explicitly permits local leader discovery; it is not authentication. Keep these ports private. Three Meta voters need a majority of two; data replication is asynchronous, so automatic failover does not promise zero data loss.

1. Install exactly v0.1.0-beta.1

Use Linux 6.1+ with usable io_uring and Ubuntu 24.04-compatible glibc. On a Mac or Windows machine, use a Linux VM. Minimal is the simplest option; Standard works with these same kernel TCP / io_uring settings. This guide does not configure SPDK devices. Allow at least 4 GiB RAM for the HA lab and additional memory for monitoring.

Download packages and SHA-256 checksums →

Use the linked package selector and installation instructions, then stay in the extracted directory. Use Bash and the same terminal for subsequent blocks; stop and inspect any failed command. On Ubuntu, install redis-tools, curl and python3; Python is used for guarded recovery checks. No global PATH change or systemd installation is required.

Package selector and copyable installation commands →

./lavik --version
./lavik-meta --version
./lavik-ctl --version

All three outputs must identify 0.1.0-beta.1. lavik-meta also prints its NuRaft revision. Do not mix nightly binaries, another Meta version, or a newer lavik-ctl with this release.

The Docker checks used a 512 MiB soft and hard locked-memory limit per process. Check your login or service memlock limits before launch; buffer-registration failures may require raising them through the service manager or login policy. In Docker, the verified limit was --ulimit memlock=536870912:536870912.

2. Create fresh state and a manifest

This creates a private directory beside the extracted binaries, one 512 MiB data file per Data process, and the complete initial manifest. It refuses an existing directory. For a real service, choose an absolute path on persistent storage before initializing; keep it independent of future package upgrades. Never point cluster-create at data you need to preserve.

# Run from the extracted v0.1.0-beta.1 package directory.
export LAVIK_BIN_DIR="$(pwd -P)"
export LAVIK_ROOT="$LAVIK_BIN_DIR/ha-beta1"
# Initialization is for a new, empty directory only.
test ! -e "$LAVIK_ROOT" || { echo "Already exists: $LAVIK_ROOT" >&2; exit 1; }
umask 077
mkdir -p "$LAVIK_ROOT"
for i in 1 2 3; do mkdir "$LAVIK_ROOT/meta-$i"; done
for i in 1 2; do
  mkdir "$LAVIK_ROOT/data-$i"
  fallocate -l 512M "$LAVIK_ROOT/data-$i/lavik.data"
done
cat > "$LAVIK_ROOT/cluster.toml" <<'TOML'
schema_version = 1
slot_strategy = "contiguous-even"

[[meta_members]]
id = 1
raft_endpoint = "tcp://127.0.0.1:7101"
data_control_endpoint = "tcp://127.0.0.1:7301"
ctl_endpoint = "tcp://127.0.0.1:7201"

[[meta_members]]
id = 2
raft_endpoint = "tcp://127.0.0.1:7102"
data_control_endpoint = "tcp://127.0.0.1:7302"
ctl_endpoint = "tcp://127.0.0.1:7202"

[[meta_members]]
id = 3
raft_endpoint = "tcp://127.0.0.1:7103"
data_control_endpoint = "tcp://127.0.0.1:7303"
ctl_endpoint = "tcp://127.0.0.1:7203"

[[data_nodes]]
id = "1111111111111111111111111111111111111111"
client_endpoint = "tcp://127.0.0.1:6371"

[[data_nodes]]
id = "2222222222222222222222222222222222222222"
client_endpoint = "tcp://127.0.0.1:6372"

[[groups]]
id = "group-1"
primary = "1111111111111111111111111111111111111111"
replicas = ["2222222222222222222222222222222222222222"]
TOML

Node IDs, Meta IDs, endpoints and Group membership must match the startup arguments. The initial primary is node 111…111. Each listed node ID has exactly 40 hexadecimal characters. The manifest allocates all slots to group-1.

3. Start Meta, then Data

The following block starts background processes, waits for a Meta leader, and then starts Data. Each process gets its own logs, state directory and PID file. The small worker, memory and file settings are for this lab, not the published performance benchmark.

# Reuse these paths after a restart; do not initialize the directory again.
: "${LAVIK_BIN_DIR:?Set the absolute package directory}"
: "${LAVIK_ROOT:?Set the existing state directory}"
test -f "$LAVIK_ROOT/cluster.toml"
for pidfile in "$LAVIK_ROOT"/*.pid; do
  [ -f "$pidfile" ] || continue
  if kill -0 "$(cat "$pidfile")" 2>/dev/null; then
    echo "A recorded process is still running: $pidfile" >&2
    exit 1
  fi
done
for i in 1 2 3; do
  bootstrap=()
  if [ ! -f "$LAVIK_ROOT/meta-$i/cluster_config.dat" ]; then
    bootstrap=(--initial-cluster-manifest "$LAVIK_ROOT/cluster.toml")
  fi
  nohup "$LAVIK_BIN_DIR/lavik-meta" --id "$i" \
    --addr "127.0.0.1:$((7100+i))" \
    --ctl-addr "127.0.0.1:$((7200+i))" \
    --data-control-addr "127.0.0.1:$((7300+i))" \
    --data-dir "$LAVIK_ROOT/meta-$i" "${bootstrap[@]}" \
    > "$LAVIK_ROOT/meta-$i.log" 2>&1 < /dev/null &
  echo "$!" > "$LAVIK_ROOT/meta-$i.pid"
done
# Wait for a leader before submitting the creation request.
deadline=$((SECONDS+30))
leader=0
while (( SECONDS < deadline )); do
  for directory in "$LAVIK_ROOT"/meta-*; do
    [ -d "$directory" ] || continue
    output=$("$LAVIK_BIN_DIR/lavik-ctl" --socket "$directory/meta-admin.sock" status 2>/dev/null) || continue
    if [[ "$output" == *"leader=1"* ]]; then leader=1; break; fi
  done
  (( leader == 1 )) && break
  sleep 1
done
(( leader == 1 )) || { echo "No Meta leader: inspect $LAVIK_ROOT/meta-*.log" >&2; exit 1; }

for i in 1 2; do
  node_id=$(printf '%040d' 0 | tr '0' "$i")
  nohup "$LAVIK_BIN_DIR/lavik" --bind 127.0.0.1 \
    --port "$((6370+i))" --metrics-port "$((9100+i))" \
    --network kernel --storage uring --threads 1 --no-pin-workers \
    --registered-buffer-mb-per-worker 64 --max-memory 512MiB \
    --shutdown-checkpoint --cluster-enabled --cluster-node-id "$node_id" \
    --cluster-announce-ip 127.0.0.1 \
    --cluster-meta-seed 127.0.0.1:7301 \
    --cluster-meta-seed 127.0.0.1:7302 \
    --cluster-meta-seed 127.0.0.1:7303 \
    --data-file "$LAVIK_ROOT/data-$i/lavik.data" \
    --log-dir "$LAVIK_ROOT/data-$i/logs" \
    > "$LAVIK_ROOT/data-$i.log" 2>&1 < /dev/null &
  echo "$!" > "$LAVIK_ROOT/data-$i.pid"
done

Before creation, a running Data process can answer LOADING because Meta has not initialized its population and authority. Check the process logs if Meta does not elect a leader. The initial manifest is passed only before cluster_config.dat exists.

4. Initialize with lavik-ctl and wait for readiness

Keep all three initial Meta members online during creation. The bootstrap workflow waits for every initial Meta state machine to catch up at its creation barrier; a majority alone does not complete that phase. Normal control-plane mutations use majority completion after it advances.

cluster-create is destructive initialization, not a restart or an attach command. Review the manifest first. --yes accepts the normalized plan without prompting; omit it if you prefer to type yes interactively. Run this once against this fresh environment.

# Destructive, one-time initialization of the fresh Data files above.
# --yes accepts the displayed plan; omit it to confirm interactively.
"$LAVIK_BIN_DIR/lavik-ctl" cluster-create \
  --manifest "$LAVIK_ROOT/cluster.toml" \
  --socket "$LAVIK_ROOT/meta-1/meta-admin.sock" \
  --allow-plaintext-admin --yes

An exit code of 0 means the Genesis request committed, not that clients can already use the cluster. Follow readiness separately. If creation exits 3, the result is uncertain: inspect cluster-status and the printed operation ID instead of submitting another create.

# cluster-create returning 0 means accepted, not ready to serve.
# Retry status reads only; never automatically retry cluster-create.
deadline=$((SECONDS+120))
ready=0
while (( SECONDS < deadline )); do
  if "$LAVIK_BIN_DIR/lavik-ctl" cluster-status \
    --socket "$LAVIK_ROOT/meta-1/meta-admin.sock" \
    --allow-plaintext-admin; then
    ready=1
    break
  else
    code=$?
    if (( code != 2 && code != 3 )); then exit "$code"; fi
  fi
  sleep 1
done
(( ready == 1 )) || { echo "Not ready: inspect status and logs; do not recreate." >&2; exit 1; }

"$LAVIK_BIN_DIR/lavik-ctl" cluster-status \
  --socket "$LAVIK_ROOT/meta-1/meta-admin.sock" \
  --allow-plaintext-admin --json

Expect cluster_state=created, result=ready, serving_ready=true and an empty blockers list. cluster-status exits 0 for READY, 2 for NOT READY, 3 for RETRYABLE, and 1 for a fatal local/transport error. Read status_explanation and next_action when it is not ready.

5. Check the follower and exercise clients

Do not use the creation acknowledgement or a single readiness snapshot as proof that the follower has finished synchronizing. For this initial layout, node 2 is the follower. Wait for its replication link and full synchronization before testing a handoff.

# Initial creation only: node 2 is the follower. After failover, discover roles again.
deadline=$((SECONDS+120))
synced=0
while (( SECONDS < deadline )); do
  if ! info=$(redis-cli -h 127.0.0.1 -p 6372 --raw INFO replication 2>/dev/null); then
    sleep 1
    continue
  fi
  if [[ "$info" == *"lavik_replication_failed_stopped:1"* ]]; then
    printf '%s\n' "$info" >&2
    echo "Follower replication stopped; inspect logs before a state-preserving follower restart." >&2
    break
  fi
  if [[ "$info" == *"master_link_status:up"* && "$info" == *"master_sync_in_progress:0"* ]]; then
    synced=1
    printf '%s\n' "$info"
    break
  fi
  sleep 1
done
if (( synced != 1 )); then
  printf '%s\n' "$info" >&2
  echo "Follower is not synchronized; inspect its log and Meta observations." >&2
fi
# Return a failed check without explicitly exiting a reader's interactive shell.
(( synced == 1 ))

Check follower synchronization independently even when cluster-status reports READY. If the previous check reports lavik_replication_failed_stopped:1 or times out with the follower link down, inspect its log. The following guarded block restarts only the initial follower without deleting its state. Run it only before the first failover, without concurrent topology changes, then repeat the follower check. If it remains blocked, preserve the files and investigate; do not recreate the cluster. The guard requires exit 0, a serving original primary in Group term 1, fresh node observations and a healthy failover detector. If it rejects the restart, inspect status and logs first; initial leadership warmup may need a few seconds. This snapshot is not a lock: keep topology administration quiescent while running it.

# Only for node 2 in the initial two-Data lab, before any failover.
# Preserve its data file, node ID and Meta state. Requires python3.
(
if ! status=$("$LAVIK_BIN_DIR/lavik-ctl" cluster-status \
  --socket "$LAVIK_ROOT/meta-1/meta-admin.sock" --allow-plaintext-admin --json); then
  echo "Cluster status is not ready; stop and inspect." >&2
  exit 1
fi
printf '%s' "$status" | python3 -c '
import json,sys
s=json.load(sys.stdin)
assert s["result"] == "ready" and s["cluster_state"] == "created"
assert all(s[k] is True for k in ("meta_available", "meta_membership_stable", "topology_converged", "serving_ready", "cluster_ready"))
assert s["blockers"] == [] and len(s["groups"]) == 1
g = s["groups"][0]
assert g["group_id"] == "group-1" and g["term"] == "1"
assert g["owner_node_id"] == "1"*40 and g["serving_ready"] is True
assert g["topology_converged"] is True and g["automatic_failover_state"] == "healthy"
assert g["current_reason"] is None and g["blocked_reason"] is None
assert len(s["data_nodes"]) == 2
for node_id, role in (("1"*40, "primary"), ("2"*40, "replica")):
    n = next(n for n in s["data_nodes"] if n["node_id"] == node_id)
    assert n["role"] == role and n["group_id"] == "group-1" and n["retired"] is False
    assert all(n[k] is True for k in ("current_session", "projection_current", "health_fresh", "population_current"))
' || { echo "Initial serving roles, term or transition checks failed; stop and inspect." >&2; exit 1; }
pid=$(cat "$LAVIK_ROOT/data-2.pid")
test "$(readlink "/proc/$pid/exe")" = "$LAVIK_BIN_DIR/lavik" || exit 1
tr '\0' '\n' < "/proc/$pid/cmdline" | grep -F -- "$LAVIK_ROOT/data-2/lavik.data" > /dev/null || exit 1
kill -TERM "$pid"
deadline=$((SECONDS+60))
while [ "$(readlink "/proc/$pid/exe" 2>/dev/null)" = "$LAVIK_BIN_DIR/lavik" ]; do
  (( SECONDS < deadline )) || { echo "Follower did not stop; inspect logs." >&2; exit 1; }
  sleep 1
done
nohup "$LAVIK_BIN_DIR/lavik" --bind 127.0.0.1 --port 6372 --metrics-port 9102 \
  --network kernel --storage uring --threads 1 --no-pin-workers \
  --registered-buffer-mb-per-worker 64 --max-memory 512MiB --shutdown-checkpoint \
  --cluster-enabled --cluster-node-id 2222222222222222222222222222222222222222 \
  --cluster-announce-ip 127.0.0.1 \
  --cluster-meta-seed 127.0.0.1:7301 --cluster-meta-seed 127.0.0.1:7302 \
  --cluster-meta-seed 127.0.0.1:7303 \
  --data-file "$LAVIK_ROOT/data-2/lavik.data" --log-dir "$LAVIK_ROOT/data-2/logs" \
  >> "$LAVIK_ROOT/data-2.log" 2>&1 < /dev/null &
echo "$!" > "$LAVIK_ROOT/data-2.pid"
)

After this optional recovery, rerun the follower synchronization block above before continuing. For routine successful initialization, the recovery block is not required.

# SET and WAIT must share a connection. WAIT is not a failover/durability guarantee.
redis-cli -c -h 127.0.0.1 -p 6371 --raw <<'COMMANDS'
SET greeting "hello from Lavik"
WAIT 1 5000
GET greeting
COMMANDS
# The replica redirects this read to the primary; -c follows that redirect.
redis-cli -c -h 127.0.0.1 -p 6372 GET greeting
redis-cli -h 127.0.0.1 -p 6371 CLUSTER SLOTS

Expect OK, 1 and hello from Lavik from the first connection. WAIT 1 5000 requests one replica acknowledgement within five seconds; a lower count means the requested acknowledgement condition was not met. It neither enables synchronous replication nor guarantees durable writes or a lossless future failover. -c follows cluster redirects; a normal read sent to the follower is redirected to the primary. Beta.1 supports CLUSTER SLOTS; do not substitute CLUSTER SHARDS.

Manage and inspect with lavik-ctl

# These are local committed reads. Select the current Meta leader first.
# Check status on each member: leader=1 identifies the leader at that moment.
for directory in "$LAVIK_ROOT"/meta-*; do
  [ -d "$directory" ] || continue
  "$LAVIK_BIN_DIR/lavik-ctl" --socket "$directory/meta-admin.sock" status
done

status is a local committed view of the selected Meta member. The member reporting leader=1 is the current leader. cluster-status discovers the leader for a cluster-wide readiness view. After a leader change, rediscover it before issuing direct reads such as observations or getop.

export LAVIK_LEADER_SOCKET=''
for directory in "$LAVIK_ROOT"/meta-*; do
  [ -d "$directory" ] || continue
  output=$("$LAVIK_BIN_DIR/lavik-ctl" --socket "$directory/meta-admin.sock" status) || continue
  if [[ "$output" == *"leader=1"* ]]; then
    export LAVIK_LEADER_SOCKET="$directory/meta-admin.sock"
    break
  fi
done
: "${LAVIK_LEADER_SOCKET:?No current Meta leader; inspect quorum and logs}"
"$LAVIK_BIN_DIR/lavik-ctl" --socket "$LAVIK_LEADER_SOCKET" observations group-1
"$LAVIK_BIN_DIR/lavik-ctl" --socket "$LAVIK_LEADER_SOCKET" getpolicy lavik.automatic-uncontrolled-failover-v1
"$LAVIK_BIN_DIR/lavik-ctl" --socket "$LAVIK_LEADER_SOCKET" getpolicy lavik.authority-lease-v1
"$LAVIK_BIN_DIR/lavik-ctl" --socket "$LAVIK_LEADER_SOCKET" getpolicy lavik.candidate-recovery-v1

observations group-1 shows the reporters’ population and source-history evidence. getpolicy returns the current policy version and document. getop takes the operation ID printed by cluster-create or failover; use it on the current leader to follow the operation. These checks are different from merely checking whether a process exists.

Planned failover before maintenance

Start only after the follower is synchronized and the current primary is healthy. A controlled failover pauses mutations while the candidate catches up to the stable source frontier. Applications should handle bounded TRYAGAIN/LOADING responses and refresh cluster topology.

"$LAVIK_BIN_DIR/lavik-ctl" failover group-1 \
  --socket "$LAVIK_ROOT/meta-1/meta-admin.sock" \
  --allow-plaintext-admin --failover-timeout-ms 120000

Exit 0 means the request committed. Copy the printed operation ID, discover the current Meta leader, then query getop until it reports OK completed failover-completed. An aborted result is not a successful cutover. If the submission exits 3, resolve that same operation before submitting anything new.

# First export LAVIK_OPERATION_ID to the actual ID printed by the CLI.
# Refresh LAVIK_LEADER_SOCKET using the leader-discovery block above.
: "${LAVIK_OPERATION_ID:?Set the operation ID printed by cluster-create or failover}"
: "${LAVIK_LEADER_SOCKET:?Discover the current Meta leader first}"
"$LAVIK_BIN_DIR/lavik-ctl" --socket "$LAVIK_LEADER_SOCKET" getop "$LAVIK_OPERATION_ID"

"$LAVIK_BIN_DIR/lavik-ctl" cluster-status \
  --socket "$LAVIK_ROOT/meta-1/meta-admin.sock" \
  --allow-plaintext-admin --json

Check groups[].owner_node_id and connect to the new primary. For the first handoff in this exact lab, that is node 222…222 on 6372. The former primary can still be LOADING while it reparents; the new primary may already serve. Recheck INFO replication, observations and a same-connection SET/WAIT probe before another handoff. Never infer candidate eligibility from process count alone.

Automatic failover is already active in beta.1. The default suspicion threshold is 5,000 ms; it is a debounce policy, not an outage-duration SLA. Detection, authority expiry, elections, candidate recovery and client retries also take time. Uncontrolled recovery may lose writes acknowledged only by the failed owner. Do not use promote --accept-data-loss as an ordinary restart step.

Stop and restart without reinitializing

Record the absolute LAVIK_BIN_DIR and LAVIK_ROOT paths. In a new terminal, export those same paths before using the commands. The stop block validates each recorded process against the executable and state directory, then sends SIGTERM. It preserves all data and Meta state. This full stop intentionally interrupts the lab service.

# Linux lab process management; this does not erase Meta or Data state.
: "${LAVIK_ROOT:?Set the existing state directory}"
: "${LAVIK_BIN_DIR:?Set the absolute package directory}"
owns_pid() {
  local pid="$1" executable
  [[ "$pid" =~ ^[0-9]+$ ]] || return 1
  executable=$(readlink "/proc/$pid/exe") || return 1
  [[ "$executable" == "$LAVIK_BIN_DIR/lavik" || "$executable" == "$LAVIK_BIN_DIR/lavik-meta" ]] || return 1
  tr '\0' '\n' < "/proc/$pid/cmdline" | grep -Fq -- "$LAVIK_ROOT/"
}
# Stop Data first, then Meta. Send SIGTERM and wait instead of using kill -9.
for role in data meta; do
  for pidfile in "$LAVIK_ROOT"/"$role"-*.pid; do
    [ -f "$pidfile" ] || continue
    pid=$(cat "$pidfile")
    if owns_pid "$pid"; then kill -TERM "$pid"; fi
  done
  deadline=$((SECONDS+60))
  while :; do
    running=0
    for pidfile in "$LAVIK_ROOT"/"$role"-*.pid; do
      [ -f "$pidfile" ] || continue
      if owns_pid "$(cat "$pidfile")"; then running=1; fi
    done
    (( running == 0 )) && break
    (( SECONDS < deadline )) || { echo "Shutdown timed out; inspect logs." >&2; exit 1; }
    sleep 1
  done
  for pidfile in "$LAVIK_ROOT"/"$role"-*.pid; do
    [ ! -f "$pidfile" ] || rm "$pidfile"
  done
done

Run the start block from step 3 again, then the readiness checks from step 4. Do not rerun directory initialization, fallocate, or cluster-create. Existing Meta state must restart without --initial-cluster-manifest. Keep the same IDs, data paths and advertised endpoints.

For an unattended service, place these same process arguments under your Linux service manager, with absolute binary paths, persistent directories and an appropriate locked-memory limit. lavik-ctl is the administrative client, not the process supervisor. Use planned failover and maintenance procedures for individual production nodes instead of stopping the whole cluster.

Roles can change during recovery. Discover the current owner with cluster-status after restarting; the initial primary in cluster.toml is not a promise about the current primary.

Set up Prometheus and Grafana

The startup commands enabled metrics on 9101 and 9102. A nonzero metrics port is required. Confirm the endpoint first; it has no authentication or TLS, so keep it on loopback or a protected monitoring network.

curl -fsS http://127.0.0.1:9101/metrics
curl -fsS http://127.0.0.1:9102/metrics

Install Docker Engine and Docker Compose v2.24.4+ on this same Linux host. Fetch the monitoring directory at beta.1’s exact source commit; it provisions Prometheus v3.11.3, Grafana 13.1.0 and the Lavik Overview dashboard. This is separate from the Lavik binaries.

If you already set up monitoring while following the other guide, reuse its directory and existing .env. Change LAVIK_TARGETS with the update procedure below rather than creating new credentials for existing volumes.

# Use a new directory on the Linux host that will run monitoring.
mkdir lavik-monitoring-beta1
cd lavik-monitoring-beta1
curl -fL https://github.com/eloqdata/lavik/archive/3955b98d43b312324aa8d52775df52cfb111c0d0.tar.gz -o source.tar.gz
(
# These upstream configuration files contain no credentials.
# Containers run as other users and need readable files/traversable directories.
umask 022
tar --no-same-permissions -xzf source.tar.gz --strip-components=3 \
  lavik-3955b98d43b312324aa8d52775df52cfb111c0d0/deploy/monitoring
)

# Run inside lavik-monitoring-beta1. Keep this file private.
# Do not overwrite an existing monitoring installation's credentials.
test ! -e .env || { echo ".env already exists; edit it instead." >&2; exit 1; }
umask 077
password=$(openssl rand -hex 24)
printf '%s\n' \
  'LAVIK_TARGETS=127.0.0.1:9101,127.0.0.1:9102' \
  'GRAFANA_ADMIN_USER=admin' \
  "GRAFANA_ADMIN_PASSWORD=$password" \
  'GRAFANA_BIND_ADDRESS=127.0.0.1' \
  'PROMETHEUS_BIND_ADDRESS=127.0.0.1' > .env
unset password

The generated .env contains only this monitoring installation’s settings and a new random Grafana password. Keep it private. Read GRAFANA_ADMIN_PASSWORD locally to sign in as admin; do not paste the file into tickets or logs. Reusing existing Grafana volumes does not reset the original admin password.

The lab binds Lavik to loopback. A normal bridge-network container cannot scrape that loopback address. This Linux-only override shares the host network and binds both monitoring UIs to loopback too. It also makes the provisioned prometheus datasource name resolve to the host-network Prometheus.

# Linux Docker Engine, Compose v2.24.4+.
# Share the Linux host network to scrape this guide's loopback-only Lavik nodes.
cat > compose.local.yaml <<'YAML'
services:
  prometheus:
    network_mode: host
    ports: !reset []
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=30d
      - --web.listen-address=127.0.0.1:9090
  grafana:
    network_mode: host
    ports: !reset []
    environment:
      GF_SERVER_HTTP_ADDR: 127.0.0.1
    extra_hosts:
      - prometheus:127.0.0.1
YAML
# --quiet validates without printing the Grafana password.
docker compose --env-file .env -f compose.yaml -f compose.local.yaml config --quiet
docker compose --env-file .env -f compose.yaml -f compose.local.yaml up -d
docker compose --env-file .env -f compose.yaml -f compose.local.yaml ps

Open http://127.0.0.1:3000/d/lavik-overview/lavik-overview for Grafana. Prometheus targets are at http://127.0.0.1:9090/targets. On a remote Linux host, use an SSH tunnel to these loopback ports. The dashboard’s Lavik instance selector filters nodes; allow at least one scrape interval before expecting data.

Open the Lavik Overview dashboard →

Open Prometheus target status →

Monitor target up/down, command throughput and latency, available storage, replication health, control-session connectivity and lease expirations. Command-duration histograms exclude socket response writes and are not end-to-end application latency. The stack provides collection and dashboards; configure your own alert rules and notification routing.

up{job="lavik"}
rate(lavik_commands_total[1m])
lavik_cluster_control_connected
increase(lavik_cluster_control_lease_expirations_total[5m])

To change the node list, edit LAVIK_TARGETS in this directory’s .env and recreate only target-config. Prometheus discovers the file change within 30 seconds. In host-network mode use the lab’s 127.0.0.1 ports; for a separate monitoring host use reachable private node IPs and the original compose.yaml without compose.local.yaml. Never use a container’s own loopback as a remote node address.

# Edit LAVIK_TARGETS in this monitoring directory's .env first.
docker compose --env-file .env -f compose.yaml -f compose.local.yaml \
  up --force-recreate --exit-code-from target-config target-config
# After exit 0, allow up to 30 seconds for discovery, then check Prometheus Targets.

Stop monitoring without deleting history. Do not add -v unless you intentionally want to delete its stored data and generated targets.

# Keep named volumes, dashboards and metric history.
docker compose --env-file .env -f compose.yaml -f compose.local.yaml down

Place the HA topology on separate hosts

Do this as a separate, fresh deployment. Do not rewrite an already-created lab’s manifest to move durable endpoints. Use three stable private addresses and place the primary and follower in separate failure domains. A third Meta voter must survive either Data host failing. Clients must reach every advertised Data endpoint.

• Give every Meta member the same complete initial manifest, replacing each endpoint with this host map. Keep the 40-character Data IDs and Group membership consistent.
• On each host, start only its assigned Meta ID with --addr HOST_IP:7100, --ctl-addr HOST_IP:7200 and --data-control-addr HOST_IP:7300, plus a private persistent directory.
• On A and B, start the assigned Data ID with --bind HOST_IP --port 6379 --metrics-port 9100 --cluster-announce-ip HOST_IP. Supply all three --cluster-meta-seed addresses on port 7300. Keep each data file private to its process.
• Initialize once with lavik-ctl against a reachable Meta Admin endpoint. Allow Meta-to-Meta Raft, Data-to-Meta control, Data replication, client-to-Data and monitoring-to-metrics traffic on their respective private ports.
• Use mTLS and the beta.1 certificate-identity rules for networks you do not fully trust. Meta Raft/Data-control TLS, Admin TLS and Data replication/client TLS have separate settings; an Admin client certificate does not secure all traffic.
• Measure application retry behavior, sustained lag, recovery and failure-domain loss yourself before adopting a production availability target. The co-located Docker tests are not a multi-host, TLS or partition certification.

Pinned beta.1 Meta, TLS and membership runbook →

Configure a cluster-aware application client with both Data endpoints as seeds and let it refresh slot ownership after redirects or connection loss. Meta Admin and Data-control ports are for the control plane, not application Redis connections. Validate the exact client’s reconnect behavior in your own failure drill.

Troubleshooting and recovery boundaries

Never delete Meta state or edit its durable configuration to force recovery. cluster-create is not a repair tool. An uncleanly stopped Data node may need population rebuild or operator recovery; a process restart alone is not proof that redundancy has returned. Preserve logs and follow the pinned recovery runbook when the group remains blocked.