Start a single node with lavik-ctl

Understand the deployment

This guide uses the lavik, lavik-meta and lavik-ctl binaries shipped together in v0.1.0-beta.1. lavik-ctl administers Meta; it does not install a version, launch OS processes, or implement a start/stop service manager. First launch Meta and Data, then use cluster-create to initialize the fresh topology.

Here “single node” means one Meta voter plus one Meta-managed Data process, with one Group owning all 16,384 slots. This is not the standalone ./lavik example and it is not highly available. Losing the only Meta voter or Data process can stop service.

The single-node and HA labs reuse the same ports. Stop the earlier lab before starting the other, and keep its original state directory if you want to return to it.

All endpoints bind to 127.0.0.1. This managed node still needs Meta to grant finite write authority; running the Data process alone is not sufficient.

1. Install exactly v0.1.0-beta.1

Use Linux 6.1+ with usable io_uring and Ubuntu 24.04-compatible glibc. On a Mac or Windows machine, use a Linux VM. Minimal is the simplest option; Standard works with these same kernel TCP / io_uring settings. This guide does not configure SPDK devices. Allow at least 4 GiB RAM for the HA lab and additional memory for monitoring.

Download packages and SHA-256 checksums →

Use the linked package selector and installation instructions, then stay in the extracted directory. Use Bash and the same terminal for subsequent blocks; stop and inspect any failed command. On Ubuntu, install redis-tools, curl and python3; Python is used for guarded recovery checks. No global PATH change or systemd installation is required.

Package selector and copyable installation commands →

./lavik --version
./lavik-meta --version
./lavik-ctl --version

All three outputs must identify 0.1.0-beta.1. lavik-meta also prints its NuRaft revision. Do not mix nightly binaries, another Meta version, or a newer lavik-ctl with this release.

The Docker checks used a 512 MiB soft and hard locked-memory limit per process. Check your login or service memlock limits before launch; buffer-registration failures may require raising them through the service manager or login policy. In Docker, the verified limit was --ulimit memlock=536870912:536870912.

2. Create fresh state and a manifest

This creates a private directory beside the extracted binaries, one 512 MiB data file per Data process, and the complete initial manifest. It refuses an existing directory. For a real service, choose an absolute path on persistent storage before initializing; keep it independent of future package upgrades. Never point cluster-create at data you need to preserve.

# Run from the extracted v0.1.0-beta.1 package directory.
export LAVIK_BIN_DIR="$(pwd -P)"
export LAVIK_ROOT="$LAVIK_BIN_DIR/single-beta1"
# Initialization is for a new, empty directory only.
test ! -e "$LAVIK_ROOT" || { echo "Already exists: $LAVIK_ROOT" >&2; exit 1; }
umask 077
mkdir -p "$LAVIK_ROOT"
for i in 1; do mkdir "$LAVIK_ROOT/meta-$i"; done
for i in 1; do
  mkdir "$LAVIK_ROOT/data-$i"
  fallocate -l 512M "$LAVIK_ROOT/data-$i/lavik.data"
done
cat > "$LAVIK_ROOT/cluster.toml" <<'TOML'
schema_version = 1
slot_strategy = "contiguous-even"

[[meta_members]]
id = 1
raft_endpoint = "tcp://127.0.0.1:7101"
data_control_endpoint = "tcp://127.0.0.1:7301"
ctl_endpoint = "tcp://127.0.0.1:7201"

[[data_nodes]]
id = "1111111111111111111111111111111111111111"
client_endpoint = "tcp://127.0.0.1:6371"

[[groups]]
id = "group-1"
primary = "1111111111111111111111111111111111111111"
replicas = []
TOML

Node IDs, Meta IDs, endpoints and Group membership must match the startup arguments. The initial primary is node 111…111. Each listed node ID has exactly 40 hexadecimal characters. The manifest allocates all slots to group-1.

3. Start Meta, then Data

The following block starts background processes, waits for a Meta leader, and then starts Data. Each process gets its own logs, state directory and PID file. The small worker, memory and file settings are for this lab, not the published performance benchmark.

# Reuse these paths after a restart; do not initialize the directory again.
: "${LAVIK_BIN_DIR:?Set the absolute package directory}"
: "${LAVIK_ROOT:?Set the existing state directory}"
test -f "$LAVIK_ROOT/cluster.toml"
for pidfile in "$LAVIK_ROOT"/*.pid; do
  [ -f "$pidfile" ] || continue
  if kill -0 "$(cat "$pidfile")" 2>/dev/null; then
    echo "A recorded process is still running: $pidfile" >&2
    exit 1
  fi
done
for i in 1; do
  bootstrap=()
  if [ ! -f "$LAVIK_ROOT/meta-$i/cluster_config.dat" ]; then
    bootstrap=(--initial-cluster-manifest "$LAVIK_ROOT/cluster.toml")
  fi
  nohup "$LAVIK_BIN_DIR/lavik-meta" --id "$i" \
    --addr "127.0.0.1:$((7100+i))" \
    --ctl-addr "127.0.0.1:$((7200+i))" \
    --data-control-addr "127.0.0.1:$((7300+i))" \
    --data-dir "$LAVIK_ROOT/meta-$i" "${bootstrap[@]}" \
    > "$LAVIK_ROOT/meta-$i.log" 2>&1 < /dev/null &
  echo "$!" > "$LAVIK_ROOT/meta-$i.pid"
done
# Wait for a leader before submitting the creation request.
deadline=$((SECONDS+30))
leader=0
while (( SECONDS < deadline )); do
  for directory in "$LAVIK_ROOT"/meta-*; do
    [ -d "$directory" ] || continue
    output=$("$LAVIK_BIN_DIR/lavik-ctl" --socket "$directory/meta-admin.sock" status 2>/dev/null) || continue
    if [[ "$output" == *"leader=1"* ]]; then leader=1; break; fi
  done
  (( leader == 1 )) && break
  sleep 1
done
(( leader == 1 )) || { echo "No Meta leader: inspect $LAVIK_ROOT/meta-*.log" >&2; exit 1; }

for i in 1; do
  node_id=$(printf '%040d' 0 | tr '0' "$i")
  nohup "$LAVIK_BIN_DIR/lavik" --bind 127.0.0.1 \
    --port "$((6370+i))" --metrics-port "$((9100+i))" \
    --network kernel --storage uring --threads 1 --no-pin-workers \
    --registered-buffer-mb-per-worker 64 --max-memory 512MiB \
    --shutdown-checkpoint --cluster-enabled --cluster-node-id "$node_id" \
    --cluster-announce-ip 127.0.0.1 \
    --cluster-meta-seed 127.0.0.1:7301 \
    --data-file "$LAVIK_ROOT/data-$i/lavik.data" \
    --log-dir "$LAVIK_ROOT/data-$i/logs" \
    > "$LAVIK_ROOT/data-$i.log" 2>&1 < /dev/null &
  echo "$!" > "$LAVIK_ROOT/data-$i.pid"
done

Before creation, a running Data process can answer LOADING because Meta has not initialized its population and authority. Check the process logs if Meta does not elect a leader. The initial manifest is passed only before cluster_config.dat exists.

4. Initialize with lavik-ctl and wait for readiness

cluster-create is destructive initialization, not a restart or an attach command. Review the manifest first. --yes accepts the normalized plan without prompting; omit it if you prefer to type yes interactively. Run this once against this fresh environment.

# Destructive, one-time initialization of the fresh Data files above.
# --yes accepts the displayed plan; omit it to confirm interactively.
"$LAVIK_BIN_DIR/lavik-ctl" cluster-create \
  --manifest "$LAVIK_ROOT/cluster.toml" \
  --socket "$LAVIK_ROOT/meta-1/meta-admin.sock" \
  --allow-plaintext-admin --yes

An exit code of 0 means the Genesis request committed, not that clients can already use the cluster. Follow readiness separately. If creation exits 3, the result is uncertain: inspect cluster-status and the printed operation ID instead of submitting another create.

# cluster-create returning 0 means accepted, not ready to serve.
# Retry status reads only; never automatically retry cluster-create.
deadline=$((SECONDS+120))
ready=0
while (( SECONDS < deadline )); do
  if "$LAVIK_BIN_DIR/lavik-ctl" cluster-status \
    --socket "$LAVIK_ROOT/meta-1/meta-admin.sock" \
    --allow-plaintext-admin; then
    ready=1
    break
  else
    code=$?
    if (( code != 2 && code != 3 )); then exit "$code"; fi
  fi
  sleep 1
done
(( ready == 1 )) || { echo "Not ready: inspect status and logs; do not recreate." >&2; exit 1; }

"$LAVIK_BIN_DIR/lavik-ctl" cluster-status \
  --socket "$LAVIK_ROOT/meta-1/meta-admin.sock" \
  --allow-plaintext-admin --json

Expect cluster_state=created, result=ready, serving_ready=true and an empty blockers list. cluster-status exits 0 for READY, 2 for NOT READY, 3 for RETRYABLE, and 1 for a fatal local/transport error. Read status_explanation and next_action when it is not ready.

5. Write and read your first key

redis-cli -c -h 127.0.0.1 -p 6371 SET greeting 'hello from Lavik'
redis-cli -c -h 127.0.0.1 -p 6371 GET greeting
redis-cli -h 127.0.0.1 -p 6371 CLUSTER INFO

Expect OK and hello from Lavik, followed by cluster information. The client uses port 6371, not the standalone quick-start port 6379. Keep -c when moving this application to multiple Groups so it can follow slot redirects.

Manage and inspect with lavik-ctl

# These are local committed reads. Select the current Meta leader first.
# Check status on each member: leader=1 identifies the leader at that moment.
for directory in "$LAVIK_ROOT"/meta-*; do
  [ -d "$directory" ] || continue
  "$LAVIK_BIN_DIR/lavik-ctl" --socket "$directory/meta-admin.sock" status
done

status is a local committed view of the selected Meta member. The member reporting leader=1 is the current leader. cluster-status discovers the leader for a cluster-wide readiness view. After a leader change, rediscover it before issuing direct reads such as observations or getop.

export LAVIK_LEADER_SOCKET=''
for directory in "$LAVIK_ROOT"/meta-*; do
  [ -d "$directory" ] || continue
  output=$("$LAVIK_BIN_DIR/lavik-ctl" --socket "$directory/meta-admin.sock" status) || continue
  if [[ "$output" == *"leader=1"* ]]; then
    export LAVIK_LEADER_SOCKET="$directory/meta-admin.sock"
    break
  fi
done
: "${LAVIK_LEADER_SOCKET:?No current Meta leader; inspect quorum and logs}"
"$LAVIK_BIN_DIR/lavik-ctl" --socket "$LAVIK_LEADER_SOCKET" observations group-1
"$LAVIK_BIN_DIR/lavik-ctl" --socket "$LAVIK_LEADER_SOCKET" getpolicy lavik.automatic-uncontrolled-failover-v1
"$LAVIK_BIN_DIR/lavik-ctl" --socket "$LAVIK_LEADER_SOCKET" getpolicy lavik.authority-lease-v1
"$LAVIK_BIN_DIR/lavik-ctl" --socket "$LAVIK_LEADER_SOCKET" getpolicy lavik.candidate-recovery-v1

observations group-1 shows the reporters’ population and source-history evidence. getpolicy returns the current policy version and document. getop takes the operation ID printed by cluster-create or failover; use it on the current leader to follow the operation. These checks are different from merely checking whether a process exists.

Stop and restart without reinitializing

Record the absolute LAVIK_BIN_DIR and LAVIK_ROOT paths. In a new terminal, export those same paths before using the commands. The stop block validates each recorded process against the executable and state directory, then sends SIGTERM. It preserves all data and Meta state. This full stop intentionally interrupts the lab service.

# Linux lab process management; this does not erase Meta or Data state.
: "${LAVIK_ROOT:?Set the existing state directory}"
: "${LAVIK_BIN_DIR:?Set the absolute package directory}"
owns_pid() {
  local pid="$1" executable
  [[ "$pid" =~ ^[0-9]+$ ]] || return 1
  executable=$(readlink "/proc/$pid/exe") || return 1
  [[ "$executable" == "$LAVIK_BIN_DIR/lavik" || "$executable" == "$LAVIK_BIN_DIR/lavik-meta" ]] || return 1
  tr '\0' '\n' < "/proc/$pid/cmdline" | grep -Fq -- "$LAVIK_ROOT/"
}
# Stop Data first, then Meta. Send SIGTERM and wait instead of using kill -9.
for role in data meta; do
  for pidfile in "$LAVIK_ROOT"/"$role"-*.pid; do
    [ -f "$pidfile" ] || continue
    pid=$(cat "$pidfile")
    if owns_pid "$pid"; then kill -TERM "$pid"; fi
  done
  deadline=$((SECONDS+60))
  while :; do
    running=0
    for pidfile in "$LAVIK_ROOT"/"$role"-*.pid; do
      [ -f "$pidfile" ] || continue
      if owns_pid "$(cat "$pidfile")"; then running=1; fi
    done
    (( running == 0 )) && break
    (( SECONDS < deadline )) || { echo "Shutdown timed out; inspect logs." >&2; exit 1; }
    sleep 1
  done
  for pidfile in "$LAVIK_ROOT"/"$role"-*.pid; do
    [ ! -f "$pidfile" ] || rm "$pidfile"
  done
done

Run the start block from step 3 again, then the readiness checks from step 4. Do not rerun directory initialization, fallocate, or cluster-create. Existing Meta state must restart without --initial-cluster-manifest. Keep the same IDs, data paths and advertised endpoints.

For an unattended service, place these same process arguments under your Linux service manager, with absolute binary paths, persistent directories and an appropriate locked-memory limit. lavik-ctl is the administrative client, not the process supervisor. Use planned failover and maintenance procedures for individual production nodes instead of stopping the whole cluster.

Roles can change during recovery. Discover the current owner with cluster-status after restarting; the initial primary in cluster.toml is not a promise about the current primary.

Set up Prometheus and Grafana

The startup commands enabled metrics on 9101. A nonzero metrics port is required. Confirm the endpoint first; it has no authentication or TLS, so keep it on loopback or a protected monitoring network.

curl -fsS http://127.0.0.1:9101/metrics

Install Docker Engine and Docker Compose v2.24.4+ on this same Linux host. Fetch the monitoring directory at beta.1’s exact source commit; it provisions Prometheus v3.11.3, Grafana 13.1.0 and the Lavik Overview dashboard. This is separate from the Lavik binaries.

If you already set up monitoring while following the other guide, reuse its directory and existing .env. Change LAVIK_TARGETS with the update procedure below rather than creating new credentials for existing volumes.

# Use a new directory on the Linux host that will run monitoring.
mkdir lavik-monitoring-beta1
cd lavik-monitoring-beta1
curl -fL https://github.com/eloqdata/lavik/archive/3955b98d43b312324aa8d52775df52cfb111c0d0.tar.gz -o source.tar.gz
(
# These upstream configuration files contain no credentials.
# Containers run as other users and need readable files/traversable directories.
umask 022
tar --no-same-permissions -xzf source.tar.gz --strip-components=3 \
  lavik-3955b98d43b312324aa8d52775df52cfb111c0d0/deploy/monitoring
)

# Run inside lavik-monitoring-beta1. Keep this file private.
# Do not overwrite an existing monitoring installation's credentials.
test ! -e .env || { echo ".env already exists; edit it instead." >&2; exit 1; }
umask 077
password=$(openssl rand -hex 24)
printf '%s\n' \
  'LAVIK_TARGETS=127.0.0.1:9101' \
  'GRAFANA_ADMIN_USER=admin' \
  "GRAFANA_ADMIN_PASSWORD=$password" \
  'GRAFANA_BIND_ADDRESS=127.0.0.1' \
  'PROMETHEUS_BIND_ADDRESS=127.0.0.1' > .env
unset password

The generated .env contains only this monitoring installation’s settings and a new random Grafana password. Keep it private. Read GRAFANA_ADMIN_PASSWORD locally to sign in as admin; do not paste the file into tickets or logs. Reusing existing Grafana volumes does not reset the original admin password.

The lab binds Lavik to loopback. A normal bridge-network container cannot scrape that loopback address. This Linux-only override shares the host network and binds both monitoring UIs to loopback too. It also makes the provisioned prometheus datasource name resolve to the host-network Prometheus.

# Linux Docker Engine, Compose v2.24.4+.
# Share the Linux host network to scrape this guide's loopback-only Lavik nodes.
cat > compose.local.yaml <<'YAML'
services:
  prometheus:
    network_mode: host
    ports: !reset []
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=30d
      - --web.listen-address=127.0.0.1:9090
  grafana:
    network_mode: host
    ports: !reset []
    environment:
      GF_SERVER_HTTP_ADDR: 127.0.0.1
    extra_hosts:
      - prometheus:127.0.0.1
YAML
# --quiet validates without printing the Grafana password.
docker compose --env-file .env -f compose.yaml -f compose.local.yaml config --quiet
docker compose --env-file .env -f compose.yaml -f compose.local.yaml up -d
docker compose --env-file .env -f compose.yaml -f compose.local.yaml ps

Open http://127.0.0.1:3000/d/lavik-overview/lavik-overview for Grafana. Prometheus targets are at http://127.0.0.1:9090/targets. On a remote Linux host, use an SSH tunnel to these loopback ports. The dashboard’s Lavik instance selector filters nodes; allow at least one scrape interval before expecting data.

Open the Lavik Overview dashboard →

Open Prometheus target status →

Monitor target up/down, command throughput and latency, available storage, replication health, control-session connectivity and lease expirations. Command-duration histograms exclude socket response writes and are not end-to-end application latency. The stack provides collection and dashboards; configure your own alert rules and notification routing.

up{job="lavik"}
rate(lavik_commands_total[1m])
lavik_cluster_control_connected
increase(lavik_cluster_control_lease_expirations_total[5m])

To change the node list, edit LAVIK_TARGETS in this directory’s .env and recreate only target-config. Prometheus discovers the file change within 30 seconds. In host-network mode use the lab’s 127.0.0.1 ports; for a separate monitoring host use reachable private node IPs and the original compose.yaml without compose.local.yaml. Never use a container’s own loopback as a remote node address.

# Edit LAVIK_TARGETS in this monitoring directory's .env first.
docker compose --env-file .env -f compose.yaml -f compose.local.yaml \
  up --force-recreate --exit-code-from target-config target-config
# After exit 0, allow up to 30 seconds for discovery, then check Prometheus Targets.

Stop monitoring without deleting history. Do not add -v unless you intentionally want to delete its stored data and generated targets.

# Keep named volumes, dashboards and metric history.
docker compose --env-file .env -f compose.yaml -f compose.local.yaml down

Troubleshooting and recovery boundaries

Never delete Meta state or edit its durable configuration to force recovery. cluster-create is not a repair tool. An uncleanly stopped Data node may need population rebuild or operator recovery; a process restart alone is not proof that redundancy has returned. Preserve logs and follow the pinned recovery runbook when the group remains blocked.