Walrus

Main WAL instance for managing topic-based write-ahead logging

The Walrus struct is the main entry point for the Walrus Write-Ahead Log library. It provides high-performance concurrent read and write operations with configurable consistency models and fsync scheduling.

Constructors

new()

Creates a new Walrus instance with default settings.

pub fn new() -> std::io::Result<Self>

Default Configuration:

• Read consistency: StrictlyAtOnce
• Fsync schedule: Milliseconds(200)
• Data directory: ./wal_files (or WALRUS_DATA_DIR env var)

Example:

use walrus_rust::Walrus;

let wal = Walrus::new()?;

with_consistency()

Creates a Walrus instance with a custom read consistency model.

pub fn with_consistency(mode: ReadConsistency) -> std::io::Result<Self>

mode · ReadConsistency · required

The read consistency model to use (StrictlyAtOnce or AtLeastOnce)

Example:

use walrus_rust::{Walrus, ReadConsistency};

let wal = Walrus::with_consistency(
    ReadConsistency::AtLeastOnce { persist_every: 1000 }
)?;

with_consistency_and_schedule()

Creates a Walrus instance with custom consistency and fsync scheduling.

pub fn with_consistency_and_schedule(
    mode: ReadConsistency,
    fsync_schedule: FsyncSchedule,
) -> std::io::Result<Self>

mode · ReadConsistency · required

The read consistency model to use

fsync_schedule · FsyncSchedule · required

When to flush data to disk (Milliseconds, SyncEach, or NoFsync)

Example:

use walrus_rust::{Walrus, ReadConsistency, FsyncSchedule};

let wal = Walrus::with_consistency_and_schedule(
    ReadConsistency::StrictlyAtOnce,
    FsyncSchedule::Milliseconds(500)
)?;

new_for_key()

Creates a namespaced Walrus instance with a separate storage directory.

pub fn new_for_key(key: &str) -> std::io::Result<Self>

key · &str · required

Namespace identifier (creates a subdirectory under wal_files/<sanitized-key>/)

Example:

use walrus_rust::Walrus;

let wal = Walrus::new_for_key("tenant-123")?;

with_consistency_for_key()

Creates a namespaced Walrus instance with custom consistency.

pub fn with_consistency_for_key(key: &str, mode: ReadConsistency) -> std::io::Result<Self>

key · &str · required

Namespace identifier

mode · ReadConsistency · required

The read consistency model to use

with_consistency_and_schedule_for_key()

Creates a namespaced Walrus instance with full custom configuration.

pub fn with_consistency_and_schedule_for_key(
    key: &str,
    mode: ReadConsistency,
    fsync_schedule: FsyncSchedule,
) -> std::io::Result<Self>

key · &str · required

Namespace identifier

mode · ReadConsistency · required

The read consistency model to use

fsync_schedule · FsyncSchedule · required

When to flush data to disk

builder()

Returns a WalrusBuilder for explicit configuration.

pub fn builder() -> WalrusBuilder

Recommended when multiple threads need to create instances with different data directories to avoid race conditions with the WALRUS_DATA_DIR environment variable.

Example:

use walrus_rust::Walrus;
use std::path::PathBuf;

let wal = Walrus::builder()
    .data_dir(PathBuf::from("/var/lib/myapp/wal"))
    .key("tenant-123")
    .build()?;

Write Operations

append_for_topic()

Appends a single entry to a topic.

pub fn append_for_topic(&self, col_name: &str, raw_bytes: &[u8]) -> std::io::Result<()>

col_name · &str · required

The topic name to append to

raw_bytes · &[u8] · required

The data bytes to append

Example:

wal.append_for_topic("my-topic", b"Hello, Walrus!")?;

batch_append_for_topic()

Atomically appends multiple entries to a topic (all-or-nothing).

pub fn batch_append_for_topic(&self, col_name: &str, batch: &[&[u8]]) -> std::io::Result<()>

col_name · &str · required

The topic name to append to

batch · &[&[u8]] · required

Slice of byte slices to append atomically (max 2,000 entries, ~10GB total)

Batch Limits:

• Maximum 2,000 entries per batch
• Maximum ~10GB total payload per batch

Performance:

• On Linux with FD backend: Uses io_uring for parallel I/O
• Other platforms or mmap backend: Sequential operations

Example:

let batch = vec![
    b"entry 1".as_slice(),
    b"entry 2".as_slice(),
    b"entry 3".as_slice(),
];
wal.batch_append_for_topic("events", &batch)?;

Read Operations

read_next()

Reads the next entry from a topic.

pub fn read_next(&self, col_name: &str, checkpoint: bool) -> std::io::Result<Option<Entry>>

col_name · &str · required

The topic name to read from

checkpoint · bool · required

• true: Consume the entry (advances read position)
• false: Peek at the entry without consuming

return · std::io::Result<Option<Entry>>

• Ok(Some(entry)): Entry was read successfully
• Ok(None): No more entries available
• Err(e): I/O error occurred

Example:

// Read and consume
if let Some(entry) = wal.read_next("my-topic", true)? {
    println!("Read: {:?}", String::from_utf8_lossy(&entry.data));
}

// Peek without consuming
if let Some(entry) = wal.read_next("my-topic", false)? {
    println!("Peeking: {:?}", String::from_utf8_lossy(&entry.data));
}

batch_read_for_topic()

Reads multiple entries from a topic up to a byte limit.

pub fn batch_read_for_topic(
    &self,
    col_name: &str,
    max_bytes: usize,
    checkpoint: bool,
    start_offset: Option<u64>,
) -> std::io::Result<Vec<Entry>>

col_name · &str · required

The topic name to read from

max_bytes · usize · required

Maximum bytes to read (returns at least 1 entry if available)

checkpoint · bool · required

• true: Consume entries (advances read position)
• false: Peek at entries without consuming

start_offset · Option<u64>

• None: Stateful read from current position
• Some(offset): Stateless read from specific byte offset

return · std::io::Result<Vec<Entry>>

Vector of entries read (may be empty if none available)

Performance:

• On Linux with FD backend: Uses io_uring for parallel I/O
• Enforces max 2,000 entries per call

Example:

// Read up to 1MB
let max_bytes = 1024 * 1024;
let entries = wal.batch_read_for_topic("events", max_bytes, true, None)?;
for entry in entries {
    println!("Read: {} bytes", entry.data.len());
}

Topic Management

mark_topic_dirty()

Marks a topic as having uncommitted changes.

pub fn mark_topic_dirty(&self, topic: &str)

topic · &str · required

The topic name to mark as dirty

mark_topic_clean()

Marks a topic as having all changes committed.

pub fn mark_topic_clean(&self, topic: &str)

topic · &str · required

The topic name to mark as clean

topic_is_clean()

Checks if a topic has uncommitted changes.

pub fn topic_is_clean(&self, topic: &str) -> bool

topic · &str · required

The topic name to check

return · bool

true if topic is clean, false if dirty

get_topic_entry_count()

Gets the number of unread entries for a topic.

pub fn get_topic_entry_count(&self, topic: &str) -> u64

topic · &str · required

The topic name to query

return · u64

Number of entries available to read (0 if none or topic doesn’t exist)

get_topic_entry_counts()

Gets entry counts for all topics.

pub fn get_topic_entry_counts(&self) -> HashMap<String, u64>

return · HashMap<String, u64>

Map of topic names to their unread entry counts

get_topic_size()

Gets the total byte size of a topic (sealed + active blocks).

pub fn get_topic_size(&self, topic: &str) -> u64

topic · &str · required

The topic name to query

return · u64

Total bytes used by the topic (0 if topic doesn’t exist)

Helper Functions

topic_entry_count()

Returns the entry count for a specific topic.

pub fn topic_entry_count(wal: &Walrus, topic: &str) -> u64

wal · &Walrus · required

Reference to a Walrus instance

topic · &str · required

The topic name

return · u64

Number of entries written to the topic

Example:

use walrus_rust::{Walrus, topic_entry_count};

let wal = Walrus::new()?;
wal.append_for_topic("events", b"data")?;

let count = topic_entry_count(&wal, "events");
println!("Entry count: {}", count);

topic_entry_counts()

Returns entry counts for all topics.

pub fn topic_entry_counts(wal: &Walrus) -> HashMap<String, u64>

wal · &Walrus · required

Reference to a Walrus instance

return · HashMap<String, u64>

Map of topic names to their entry counts

Example:

use walrus_rust::{Walrus, topic_entry_counts};

let wal = Walrus::new()?;
let counts = topic_entry_counts(&wal);
for (topic, count) in counts {
    println!("{}: {} entries", topic, count);
}

Module Functions

These functions control global backend behavior and must be called before creating any Walrus instances.

enable_fd_backend()

Enables the FD (file descriptor) backend with pread/pwrite (default).

use walrus_rust::enable_fd_backend;

enable_fd_backend();
let wal = Walrus::new()?;

Features:

• Uses io_uring on Linux for batch operations
• Opens files with O_SYNC when FsyncSchedule::SyncEach is used

disable_fd_backend()

Disables the FD backend and uses memory-mapped files instead.

use walrus_rust::disable_fd_backend;

disable_fd_backend();
let wal = Walrus::new()?;

Features:

• Uses memory-mapped files
• Batch operations fall back to sequential reads/writes
• Works on all platforms

Related Types

• Entry (WalEntry) - Structure returned by read operations
• ReadConsistency - Consistency model configuration
• FsyncSchedule - Fsync timing configuration