JuliaDB

This repo contains a database toy implementation, meant to experiment with concepts described in the book Designing Data-Intensive Applications.

Project

Implement a LSM-Tree containing:

• SET method.
• GET method.
• DELETE method.
• Memtable.
• SSTable.
• flush_memtable method.
• WAL (Write Ahead Log) SET method.
• WAL (Write Ahead Log) DELETE method.
• WAL (Write Ahead Log) recover method.
• SSTable compact method.
• Bloom Filter.
• gRPC server.

Implement distributed attributes:

• Multi-partition support.
• Parition selection algorithm that works for any key.
• Fault-tolerancy in case one partition is down.
• New node addition and rebalancing.

Metrics

• Latency of SET, GET and DELETE.
• Latency of gRPC requests.
• Memtable memory usage.
• SSTable memory usage.
• SSTable index memory usage.
• flush_memtable latency.
• compact sstables latency.
• Latency to return None if key not found.

Instalation

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Running

To run in server mode:

make server

To run inside a docker container with monitoring via prometheus and visualization using grafana, run:

docker compose up -d

Alternatively, you can access it via python repl:

python -i
>>> from src.lsm_tree import LSMTree()
>>> db = LSMTree()
>>> db.get("user:123")

Populate data

To populate a local database with data run:

make populate

Cleaning

You can clean all local data using:

make clean

Tests

To run the unit tests simply:

make test-unit

To run the load tests, first install the k6 tool:

# To run only the test for the SET operation.
k6 run test/load/set_load_test.js

# To run only the test for the GET operation.
k6 run test/load/get_load_test.js