Redis Server Implementation in C++17

A Production-Grade, Multi-Threaded Redis Server Built From Scratch

Demonstrates deep systems programming expertise through distributed systems implementation

View Demo • Features • Architecture • Technical Details

---

🎯 Project Overview

This is a fully-functional Redis server implementation written in modern C++17, built entirely from scratch without using any Redis libraries. It implements the complete Redis Serialization Protocol (RESP), supports distributed master-replica replication with synchronous acknowledgment, and handles persistent storage through RDB files.

Why This Project Matters

As a portfolio project for technical roles, this demonstrates:

• ✅ Low-level systems programming with raw POSIX sockets and manual memory management
• ✅ Distributed systems design including replication, consensus, and consistency models
• ✅ Advanced concurrency using thread-per-client model with fine-grained locking
• ✅ Protocol implementation with custom streaming parser handling TCP fragmentation
• ✅ Production-ready code with 90+ incremental stages, modular architecture, and comprehensive testing

---

🚀 Core Capabilities

Data Structures & Commands

Strings SET / GET with millisecond expiry INCR for atomic counters DEL, KEYS, TYPE for management Lists RPUSH, LPUSH for queue/stack operations LRANGE with positive/negative indexing BLPOP with timeout (blocking I/O) Streams XADD with auto-generated IDs XRANGE for time-series queries XREAD with blocking reads

Sorted Sets ZADD, ZRANGE for leaderboards ZRANK, ZSCORE for lookups ZCARD, ZREM for management Geospatial GEOADD for coordinate storage GEOPOS for position retrieval Distance and radius calculations Transactions MULTI / EXEC / DISCARD Per-client command queuing Atomic execution guarantees

Advanced Features

Feature	Implementation	Technical Highlight
Pub/Sub	SUBSCRIBE, PUBLISH, UNSUBSCRIBE	Isolated subscriber mode with channel multiplexing
Replication	Master-Replica with PSYNC handshake	Asynchronous propagation + synchronous WAIT
Persistence	RDB file loading on startup	Binary format parsing with expiry restoration
Authentication	ACL system with AUTH command	SHA-256 password hashing

---

🏗 System Architecture

Thread-Per-Client Concurrency Model

┌─────────────────────────────────────────────────────────────────┐
│                         Redis Server                            │
│                                                                 │
│  ┌──────────────┐      ┌──────────────┐      ┌──────────────┐   │
│  │   Client 1   │      │   Client 2   │      │   Client N   │   │
│  │ std::thread  │      │ std::thread  │      │ std::thread  │   │
│  └──────┬───────┘      └──────┬───────┘      └──────┬───────┘   │
│         │                     │                     │           │
│         │  Fine-Grained Locks │                     │           │
│         └─────────────────────┼─────────────────────┘           │
│                               ▼                                 │
│                 ┌─────────────────────────────┐                 │
│                 │   Shared Resources          │                 │
│                 ├─────────────────────────────┤                 │
│                 │ • kv_store (std::mutex)     │                 │
│                 │ • replicas (std::mutex)     │                 │
│                 │ • pubsub_channels (mutex)   │                 │
│                 │ • Condition Variables       │                 │
│                 └─────────────────────────────┘                 │
└─────────────────────────────────────────────────────────────────┘

Key Design Decisions:

1. Thread-Per-Client: Each TCP connection spawns a detached std::thread for natural isolation of client state (transactions, subscriptions, blocking operations)

2. Fine-Grained Locking: Separate mutexes for different resources instead of a global lock, maximizing concurrency:

   • kv_store_mutex for key-value operations
   • replicas_mutex for replication state
   • pubsub_mutex for subscription management

3. Condition Variables: Used for efficient blocking operations:

   • BLPOP: Threads sleep until list has elements
   • WAIT: Threads sleep until replicas acknowledge
   • XREAD: Threads sleep until stream entries arrive

Master-Replica Replication Architecture

                         ┌──────────────────────┐
                         │       MASTER         │
                         │                      │
                         │  master_repl_id      │
                         │  master_repl_offset  │
                         └──────────┬───────────┘
                                    │
                    ┌───────────────┼───────────────┐
                    │               │               │
                    ▼               ▼               ▼
            ┌───────────┐   ┌───────────┐   ┌───────────┐
            │ Replica 1 │   │ Replica 2 │   │ Replica N │
            │           │   │           │   │           │
            │ offset: X │   │ offset: Y │   │ offset: Z │
            └───────────┘   └───────────┘   └───────────┘

Replication Flow:

1. Handshake (3-way):

   Replica → Master: PING
   Replica → Master: REPLCONF listening-port <port>
   Replica → Master: REPLCONF capa psync2
   Replica → Master: PSYNC ? -1
   Master → Replica: FULLRESYNC <repl_id> <offset>
   Master → Replica: <RDB file bytes>
   

2. Command Propagation:

   • Master forwards all write commands (SET, DEL, etc.) to replicas
   • Commands serialized in RESP format
   • Sent asynchronously to prevent client latency

3. Offset Tracking:

   • Master tracks global master_repl_offset (bytes propagated)
   • Each replica tracks bytes_processed (bytes applied)
   • Periodic health checks via REPLCONF GETACK *

4. Synchronous Replication (WAIT):

   • Client blocks until N replicas acknowledge
   • Implemented using condition variables for efficiency

---

s

📊 Performance Characteristics

Operation	Time Complexity	Concurrency Model
GET / SET	O(1)	Lock per access
LPUSH / RPUSH	O(1)	Per-list lock
LRANGE	O(N)	Read lock (allows concurrent reads)
ZADD	O(log N)	Per-zset lock
XADD	O(1)	Per-stream lock
Replication Propagation	O(1) amortized	Asynchronous, non-blocking

Scalability: With fine-grained locking, the server can handle 1000+ concurrent clients limited primarily by OS thread limits and memory, not lock contention.

---

🛠 Building & Running

Prerequisites

# Required
- C++17 compiler (GCC 7+, Clang 5+, MSVC 2017+)
- CMake 3.10+
- POSIX-compliant OS (Linux, macOS, WSL)

Quick Start

# 1. Clone repository
git clone https://github.com/codev-aryan/redis-server-implementation.git
cd redis-server-implementation

# 2. Build
mkdir build && cd build
cmake ..
cmake --build .

# 3. Run as standalone master
./your_program.sh

# 4. In another terminal, test with redis-cli
redis-cli -p 6379
127.0.0.1:6379> SET hello world
OK
127.0.0.1:6379> GET hello
"world"

Advanced Usage

Run as Replica:

# Terminal 1: Start master on port 6379
./your_program.sh --port 6379

# Terminal 2: Start replica
./your_program.sh --port 6380 --replicaof localhost 6379

# Terminal 3: Test replication
redis-cli -p 6379
127.0.0.1:6379> SET replicated_key "this will sync"
OK

redis-cli -p 6380
127.0.0.1:6380> GET replicated_key
"this will sync"  # ✅ Replicated successfully

With Persistence:

./your_program.sh --dir /tmp/redis --dbfilename dump.rdb
# Server loads existing RDB file on startup

Test Synchronous Replication:

# With 2 replicas running
redis-cli -p 6379
127.0.0.1:6379> SET x 100
OK
127.0.0.1:6379> WAIT 2 5000  # Wait for 2 replicas, 5sec timeout
(integer) 2  # Both replicas acknowledged ✅

---

📁 Project Structure

redis-server-implementation/
├── src/
│   ├── commands/              # Command handlers (modular design)
│   │   ├── cmd_strings.cpp    # SET, GET, INCR
│   │   ├── cmd_lists.cpp      # RPUSH, LPOP, BLPOP
│   │   ├── cmd_stream.cpp     # XADD, XRANGE, XREAD
│   │   ├── cmd_tx.cpp         # MULTI, EXEC, DISCARD
│   │   ├── cmd_replication.cpp # PSYNC, REPLCONF, WAIT
│   │   ├── cmd_pubsub.cpp     # SUBSCRIBE, PUBLISH
│   │   ├── cmd_zset.cpp       # ZADD, ZRANGE, ZSCORE
│   │   ├── cmd_geo.cpp        # GEOADD, GEOPOS, GEOSEARCH
│   │   ├── cmd_auth.cpp       # ACL, AUTH
│   │   └── dispatcher.cpp     # Command routing
│   │
│   ├── db/                    # Data layer
│   │   ├── database.cpp       # Core key-value store
│   │   ├── rdb_loader.cpp     # RDB binary format parser
│   │   └── structs/           # Data structure implementations
│   │       ├── redis_list.hpp
│   │       ├── redis_stream.hpp
│   │       ├── redis_string.hpp
│   │       └── redis_zset.hpp
│   │
│   ├── protocol/
│   │   └── parser.cpp         # RESP streaming parser
│   │
│   ├── server/
│   │   ├── server.cpp         # Main event loop, socket handling
│   │   └── client.cpp         # Per-client state management
│   │
│   ├── utils/
│   │   ├── geohash.cpp        # Geospatial encoding (base32)
│   │   └── sha256.cpp         # Cryptographic hashing
│   │
│   └── main.cpp               # Entry point, argument parsing
│
├── CMakeLists.txt             # Build configuration
└── README.md                  # This file

Design Principles:

• Separation of Concerns: Commands, protocol, storage, and networking are isolated
• Single Responsibility: Each .cpp file handles one domain
• Extensibility: New commands require only adding to commands/ and updating dispatcher

---

🎓 Key Learning Outcomes

This project demonstrates mastery of:

Systems Programming

• Raw socket programming with socket(), bind(), listen(), accept()
• Manual buffer management and TCP stream handling
• POSIX threading with pthread (via std::thread)

Distributed Systems

• Replication protocols: Handshake, sync, propagation, acknowledgment
• Consistency models: Eventual consistency (async replication) vs strong consistency (WAIT)
• Failure handling: Replica disconnection, partial failures
• CAP theorem trade-offs: Availability vs Consistency decisions

Concurrency & Parallelism

• Thread-per-client vs thread pool models (chose thread-per-client for simplicity)
• Fine-grained locking to minimize contention
• Proper use of std::condition_variable for blocking without CPU waste
• Deadlock avoidance through lock ordering

Software Engineering

• Incremental development (90+ stages from basic PING to full replication)
• Modular architecture enabling independent testing
• Clean code with clear abstractions
• Version control with Git, proper commit history

---

🏆 Implementation Milestones

This server was built through 90+ progressive stages, each adding complexity:

Stage	Feature	Technical Challenge
1-5	TCP binding, PING/PONG	Socket programming basics
6-15	RESP parser, ECHO, SET/GET	Protocol implementation, string handling
16-25	Expiry, concurrent clients	Timers, multi-threading
26-40	Lists, blocking operations	Condition variables, deadlock prevention
41-55	Streams, Transactions	Complex data structures, atomicity
56-70	Replication handshake	State machines, binary protocol
71-80	Command propagation, WAIT	Distributed consensus, offset tracking
81-90	Pub/Sub, Auth, Geospatial	Advanced features, cryptography

Each stage required passing automated tests before proceeding, ensuring correctness at every step.

---

🔮 Future Enhancements

Potential extensions demonstrating additional expertise:

• Redis Cluster: Consistent hashing, slot migration, CLUSTER commands
• AOF Persistence: Append-only file for durability, background rewriting
• Event-Driven I/O: Migrate to epoll (Linux) or kqueue (BSD/macOS) for better scalability
• Lock-Free Structures: Use atomic operations for high-contention paths
• Lua Scripting: Embed Lua for server-side computation (EVAL / EVALSHA)
• Compression: LZF compression for RDB/replication stream
• Monitoring: INFO command with metrics, slow log

---

📞 Contact & Links

Developer: Aryan Mehta
Repository: github.com/codev-aryan/redis-server-implementation
LinkedIn: Connect with me

---

📄 License

MIT License - feel free to use this code for learning or as reference.

---

Built with passion for systems programming and distributed systems

⭐ Star this repo if you find it impressive! ⭐

Report Bug • Request Feature • View Documentation