Configuration

Configure Walrus distributed log streaming engine with CLI flags and environment variables

Walrus provides flexible configuration through command-line flags and environment variables. This guide covers all available configuration options for both distributed and standalone deployments.

Distributed Walrus Configuration

CLI Flags

When running a distributed Walrus node, use these command-line flags to configure the cluster:

Flag	Default	Description
--node-id	(required)	Unique node identifier (integer)
--data-dir	./data	Root directory for storage
--raft-port	6000	Raft/Internal RPC port
--raft-host	127.0.0.1	Raft bind address
--raft-advertise-host	(raft-host)	Advertised Raft address for peer discovery
--client-port	8080	Client TCP port
--client-host	127.0.0.1	Client bind address
--join	-	Address of existing node to join (e.g., 127.0.0.1:6001)
--initial-peer	-	List of initial peer addresses for bootstrapping leader
--log-file	-	Optional file path for log output (defaults to stdout)

Starting a 3-Node Cluster

Node 1 (Bootstrap Leader)

cargo run -- \
  --node-id 1 \
  --raft-port 6001 \
  --client-port 9091 \
  --raft-host 127.0.0.1 \
  --client-host 127.0.0.1

The first node (node-id 1) without a --join flag automatically becomes the bootstrap leader. All subsequent nodes must use --join to connect to an existing cluster member.

Docker Deployment

When running Walrus in Docker containers, bind to 0.0.0.0 and use advertised addresses:

Docker Example

docker run -p 9091:9091 -p 6001:6001 walrus \
  --node-id 1 \
  --raft-host 0.0.0.0 \
  --raft-advertise-host walrus-1.example.com \
  --client-host 0.0.0.0 \
  --data-dir /data

Environment Variables

Configure runtime behavior with these environment variables:

Segment Configuration

WALRUS_MAX_SEGMENT_ENTRIES · integer · default: "1000000"

Number of entries before automatic segment rollover. Higher values reduce rollover frequency but increase segment size.

export WALRUS_MAX_SEGMENT_ENTRIES=2000000

WALRUS_MONITOR_CHECK_MS · integer · default: "10000"

Monitor loop interval in milliseconds. Controls how frequently the system checks for segment rollover conditions.

export WALRUS_MONITOR_CHECK_MS=5000  # Check every 5 seconds

Storage Backend Configuration

WALRUS_DISABLE_IO_URING · boolean · default: "false"

Disable io_uring backend on Linux and use mmap instead. Set to any value to disable io_uring.

export WALRUS_DISABLE_IO_URING=1

Disabling io_uring will reduce throughput for batch operations on Linux. Only disable if you encounter compatibility issues.

WALRUS_DATA_DIR · string · default: "wal_files"

Base directory for WAL storage files (standalone library usage).

export WALRUS_DATA_DIR=/var/lib/walrus/data

WALRUS_INSTANCE_KEY · string

Namespace isolation key for standalone library usage. Creates a subdirectory under WALRUS_DATA_DIR.

export WALRUS_INSTANCE_KEY=tenant-123

Logging Configuration

RUST_LOG · string · default: "info"

Log level for the application. Valid values: trace, debug, info, warn, error.

export RUST_LOG=debug
export RUST_LOG=walrus=debug,info  # Module-specific logging

WALRUS_QUIET · boolean

Suppress debug output from the storage layer. Set to any value to enable quiet mode.

export WALRUS_QUIET=1

Storage Engine Configuration

When using Walrus as a standalone library, configure these parameters via the API:

Consistency Models

use walrus_rust::{Walrus, ReadConsistency};

// Strict consistency - every read checkpoint is persisted immediately
let wal = Walrus::with_consistency(ReadConsistency::StrictlyAtOnce)?;

// At-least-once delivery - persist every N reads (higher throughput)
let wal = Walrus::with_consistency(
    ReadConsistency::AtLeastOnce { persist_every: 1000 }
)?;

Fsync Scheduling

use walrus_rust::{Walrus, FsyncSchedule};

// Fsync every 500ms (default is 200ms)
let wal = Walrus::with_consistency_and_schedule(
    ReadConsistency::StrictlyAtOnce,
    FsyncSchedule::Milliseconds(500)
)?;

// Fsync after every single write (maximum durability)
let wal = Walrus::with_consistency_and_schedule(
    ReadConsistency::StrictlyAtOnce,
    FsyncSchedule::SyncEach
)?;

// Never fsync (maximum throughput, no durability)
let wal = Walrus::with_consistency_and_schedule(
    ReadConsistency::StrictlyAtOnce,
    FsyncSchedule::NoFsync
)?;

Storage Layout

Walrus organizes data in the following directory structure:

data/
├── node_1/
│   ├── user_data/          # Walrus storage engine files
│   │   └── data_plane/     # Namespace for user data
│   │       ├── 1234567890  # Timestamp-based WAL files (1GB each)
│   │       └── 1234567891
│   └── raft_meta/          # Raft metadata storage
│       ├── raft.db
│       └── snapshots/
├── node_2/
│   └── ...
└── node_3/
    └── ...

File Characteristics

• WAL Files: Pre-allocated to 1GB (100 × 10MB blocks)
• Naming: Millisecond timestamp (monotonically increasing)
• Block Size: 10MB logical segments
• Entry Format: 64-byte metadata prefix + payload + FNV-1a checksum

Configuration Best Practices

Production Deployment

• Use at least 3 nodes for fault tolerance
• Set WALRUS_MAX_SEGMENT_ENTRIES based on your write rate and rollover frequency needs
• Keep default WALRUS_MONITOR_CHECK_MS (10s) unless you need faster rollover detection
• Use --log-file for persistent logging in production
• Mount --data-dir on a fast SSD with adequate space

High-Throughput Workloads

• Ensure io_uring is enabled (default on Linux)
• Use FsyncSchedule::Milliseconds(500) or higher for write-heavy workloads
• Consider ReadConsistency::AtLeastOnce with persist_every: 10000 for maximum read throughput
• Increase WALRUS_MAX_SEGMENT_ENTRIES to reduce rollover overhead

Development and Testing

• Single node is sufficient for local development
• Use RUST_LOG=debug for detailed logs
• Smaller WALRUS_MAX_SEGMENT_ENTRIES (e.g., 10000) for faster rollover testing
• Set WALRUS_MONITOR_CHECK_MS=1000 for rapid rollover during tests

Configuration Validation

On startup, Walrus validates configuration and logs warnings for potential issues:

# Example startup output
INFO  Node 1 booting
INFO  Registered node address 127.0.0.1:6001
INFO  Monitor loop started
INFO  Client listener bound to 127.0.0.1:9091
INFO  Lease update loop started

If all components start successfully, your configuration is valid. Check logs for any ERROR or WARN messages.

Next Steps

Monitoring

Learn how to monitor cluster health and metrics

Performance Tuning

Optimize Walrus for your workload