FerrumKV

What I built

A 24-module (~19,200 LOC) Redis-compatible data store and distributed systems engine written from scratch in safe Rust (#![forbid(unsafe_code)], zero external crates in Cargo.toml, and zero .unwrap()/.expect() calls across src/). A work-stealing TCP reactor parses zero-copy RESP3 frames alongside live Prometheus HTTP scrapes (GET /metrics) on the same socket, routing keys across 64 concurrent shards or 16,384 CRC16-CCITT cluster hash slots with -MOVED and -ASK redirections.

Beyond Robin Hood open-addressed hash tables (SipHash-2-4 / FNV-1a), circular ring lists, and level-32 sorted-set skip lists, the engine implements Redis Streams with Consumer Group Pending Entries Lists (XADD, XREADGROUP, XACK), a 16,384-register HyperLogLog, Kirsch-Mitzenmacher Bloom filters, a 4×2048 Count-Min Sketch, 52-bit Morton Z-order geohashes with WGS84 Haversine radius queries, multi-shard optimistic concurrency transactions (WATCH/MULTI/EXEC), a FIPS 180-4 SHA-256 stack bytecode VM (EVALSHA), a multi-layer HNSW vector index with i8 scalar quantization (SQ8), an Okapi BM25 full-text index with LEB128 delta postings and Damerau-Levenshtein fuzzy matching, a disk-backed LSM-tree with 4 KB CRC32-verified SSTables, Facebook Gorilla XOR time-series compression, and RFC 8439 ChaCha20-Poly1305 AEAD snapshot encryption.

What was checked

Verified with 22 unit and end-to-end TCP integration tests under rustc -D warnings, an automated 3-node TCP Raft Jepsen fault-injection harness (ferrum_kv jepsen), and a 50-client pipelined benchmark suite (ferrum_kv bench --clients 50 --requests 100000 --pipeline 16). Under concurrent leader isolation, majority failover, graceful leader step-down, and mid-syscall torn AOF writes, the cluster converged at term=3, commit_index=200 (199 committed ops, 24 rejected during partition, 0 stale/uncommitted writes acknowledged, 61 torn AOF bytes repaired) and passed Wing & Gong real-time linearizability verification. At -C opt-level=3, 50 concurrent clients achieved 505,716 req/s on GET (p50 = 39.66 µs), 424,669 req/s on MIXED (p50 = 22.84 µs), 390,313 req/s on SET, 347,136 req/s on ZADD, 274,322 req/s on EVALSHA, and 173,007 req/s on VSEARCH (Recall@5 = 93.3%).

Verification date: 8 November 2026. Benchmarks and Jepsen fault-injection tests run against in-process loopback TCP clusters on a single host and do not establish multi-datacenter WAN latency or production durability guarantees.

What this demonstrates

Demonstrates from-scratch systems programming in safe Rust across wire protocols, lock-striped data structures, Raft consensus replication, vector/full-text indexing, LSM-tree SSTable compaction, bitwise telemetry compression, and authenticated encryption without third-party dependencies.