From ce32eca20f4b5dc8b08ec933e15b4c2335e34292 Mon Sep 17 00:00:00 2001 From: Abhishek Chatterjee Date: Fri, 16 Jan 2026 22:48:15 +0530 Subject: [PATCH] IMDEEPMIND-18: Revise ACID documentation for clarity and completeness. Enhanced explanations of properties, added practical examples, and improved overall structure and readability. --- docs/databases/database-engineering/acid.md | 526 +++++++++++++++++--- 1 file changed, 445 insertions(+), 81 deletions(-) diff --git a/docs/databases/database-engineering/acid.md b/docs/databases/database-engineering/acid.md index 63ecc84a..dd4e4c78 100644 --- a/docs/databases/database-engineering/acid.md +++ b/docs/databases/database-engineering/acid.md @@ -4,134 +4,498 @@ sidebar_position: 4 # ACID -The **ACID** properties are a set of four key principles that ensure reliable processing of database transactions. ACID stands for **Atomicity**, **Consistency**, **Isolation**, and **Durability**. These properties guarantee that database transactions are processed reliably and safely, even in situations like system failures, crashes, or concurrent access by multiple users. Below is a breakdown of each property with practical examples. +:::tip[Status] + +This note is complete, reviewed, and considered stable. + +::: + +**ACID** is a set of four guarantees that define a **transaction**: + +- **A**tomicity +- **C**onsistency +- **I**solation +- **D**urability + +A **transaction** is a logical unit of work that moves the database from one valid state to another. + +
+ +```mermaid +flowchart LR + Start[Transaction Start] + Ops[Read / Write Operations] + Commit[Commit] + Abort[Rollback] + Start --> Ops + Ops --> Commit + Ops --> Abort +``` + +
+ +## Why ACID Exists + +Databases exist to store **shared, persistent state**. The moment we introduce: + +- Multiple users +- Concurrent reads & writes +- Crashes, power failures, network issues + +We face a core problem: + +> **How do we ensure the data is always correct, even when things go wrong?** + +Without strict guarantees: + +- Money can be created or lost +- Inventory can go negative +- Partial updates can corrupt data + +ACID is the **contract** between the database and our application that defines _correctness_. ## Atomicity -**Atomicity** means that a transaction is treated as a single unit of work, where either all operations within the transaction are successfully completed, or none of them are. If an error occurs or something goes wrong, the transaction must be **rolled back** to ensure the database does not remain in an inconsistent state. +> Either **all operations** in a transaction succeed, or **none of them do**. -- If any operation in the transaction fails (e.g., a `UPDATE` fails), the entire transaction will be rolled back, and no changes will be applied to the database. -- A rollback ensures the database is restored to its state before the transaction started. +There is **no partial state** visible. -**Example:** +### Example (Bank Transfer) -Consider a transaction that involves transferring money from one account to another. If any step fails, both operations (debit and credit) should be rolled back. +```text +Transfer $100 from A to B +1. A.balance -= 100 +2. B.balance += 100 +``` -```sql -BEGIN; +If the system crashes after step 1 but before step 2: + +- Without Atomicity → money disappears +- With Atomicity → both steps rollback + +### How Atomicity Is Implemented Internally + +**Write-Ahead Logging (WAL)** is the core mechanism. + +Rules: --- Debit from account 1 -UPDATE accounts SET balance = balance - 100 WHERE account_id = 1; +1. Changes are written to the **log first** +2. Only then are data pages modified --- If an error occurs, roll back the transaction --- Let's assume the credit operation fails, and We roll back -UPDATE accounts SET balance = balance + 100 WHERE account_id = 2; +
--- If something goes wrong, rollback the transaction -ROLLBACK; +```mermaid +sequenceDiagram + participant App + participant DB + participant WAL + + App->>DB: Begin Transaction + DB->>WAL: Log intended changes + DB->>DB: Modify data pages + App->>DB: Commit + DB->>WAL: Mark transaction committed ``` -In this example, if the `UPDATE` statement for the second account fails, the `ROLLBACK` command undoes both operations, ensuring that no money is debited or credited and the database stays in a consistent state. +
+ +If a crash happens: + +- Log is replayed +- Uncommitted transactions are undone + +### Common Misconception + +Atomicity ≠ single statement + +A transaction can include: + +- Multiple tables +- Multiple indexes +- Triggers +- Foreign key checks ## Consistency -**Consistency** ensures that a transaction takes the database from one **valid state** to another. All the rules, constraints, and triggers in the database (such as foreign keys, unique constraints, etc.) must be preserved during the transaction. +> A transaction must move the database from **one valid state** to **another valid state**. + +Consistency is about **rules**, not isolation or crashes. -- If any transaction operation violates the database rules (e.g., attempting to insert duplicate values into a column with a unique constraint), the transaction will be rolled back, and the database will return to its consistent state. +### Types of Consistency -**Example:** +#### Database-Level Consistency -Imagine a constraint that ensures an account balance cannot go negative. If a transaction attempts to withdraw more than the available balance, the transaction will violate this rule and be rolled back. +Enforced by the DB engine: + +- Primary keys +- Foreign keys +- NOT NULL +- UNIQUE +- CHECK constraints ```sql -BEGIN; -UPDATE accounts SET balance = balance - 100 WHERE account_id = 1; --- This will fail if the balance goes below zero, and the transaction will be rolled back -COMMIT; +balance >= 0 ``` +#### Application-Level Consistency + +Business rules: + +- User cannot withdraw more than daily limit +- Order must have at least one item + +The database **cannot enforce these automatically**. + +### Important Truth + +> **The database guarantees consistency _only if your transaction logic is correct_.** + +If you write buggy logic, ACID will faithfully preserve the bug. + ## Isolation -**Isolation** refers to the level at which the operations of one transaction are isolated from the operations of other concurrent transactions. Different **isolation levels** control the visibility of intermediate transaction states to other transactions. PostgreSQL provides several isolation levels that determine how transactions interact with one another. +> Concurrent transactions should not interfere in a way that violates correctness. -The four isolation levels are: +But **full isolation is expensive**, so databases provide **levels**. -1. **Read Uncommitted:** Allows dirty reads. Transactions can see uncommitted changes made by other transactions. -2. **Read Committed:** A transaction sees only committed data at the time it is read. It prevents dirty reads but still allows non-repeatable reads. -3. **Repeatable Read:** Ensures that once a transaction reads a value, it will see the same value throughout the transaction, preventing dirty and non-repeatable reads. Phantom reads are still possible. -4. **Serializable:** The highest isolation level, which ensures that transactions execute as if they Oure serialized (one after the other). This level prevents dirty reads, non-repeatable reads, and phantom reads. +## Isolation Problems -We can set the isolation level for a transaction in PostgreSQL using the `SET TRANSACTION ISOLATION LEVEL` command. +Isolation problems describe **anomalies that occur when multiple transactions run concurrently without sufficient isolation**. The diagrams below show _what happens_; the explanations tell you _why it is a problem_ and _what guarantee is violated_. -**Example of setting isolation levels in PostgreSQL:** +Think of isolation as answering this question: -```sql --- Start transaction with Read Committed isolation level (default) -BEGIN; -SET TRANSACTION ISOLATION LEVEL READ COMMITTED; --- Our queries here -COMMIT; - --- Start transaction with Repeatable Read isolation level -BEGIN; -SET TRANSACTION ISOLATION LEVEL REPEATABLE READ; --- Our queries here -COMMIT; - --- Start transaction with Serializable isolation level -BEGIN; -SET TRANSACTION ISOLATION LEVEL SERIALIZABLE; --- Our queries here -COMMIT; +> “If two transactions run at the same time, can one see the intermediate, incomplete, or changing results of the other?” + +### Dirty Read + +A transaction reads data written by another transaction that has **not yet committed**. + +**Why it is dangerous:** +If the writing transaction rolls back, the reading transaction has used **data that never officially existed**. + +**Real-world example:** + +- Transaction T1 updates an account balance but hasn’t committed yet. +- Transaction T2 reads this new balance and makes a business decision. +- T1 rolls back. +- T2 has now acted on **phantom data**. + +
+ +```mermaid +sequenceDiagram + participant T1 + participant T2 + + T1->>DB: Write X=100 (not committed) + T2->>DB: Read X=100 (dirty read) + T1->>DB: Rollback +``` + +
+ +**Key insight:** + +> Dirty reads violate the illusion that a transaction only sees _final, committed_ data. + +### Non-Repeatable Read + +A transaction reads the **same row twice** and gets **different values**, because another transaction modified and committed that row in between. + +**Why it is dangerous:** +The transaction cannot rely on data staying stable during its execution. + +**Real-world example:** + +- You read your bank balance. +- Another transaction updates it. +- You read it again within the same transaction and see a different value. + +
+ +```mermaid +sequenceDiagram + participant T1 + participant T2 + + T1->>DB: Read X=50 + T2->>DB: Update X=70 + Commit + T1->>DB: Read X=70 (non-repeatable) +``` + +
+ +**Key insight:** + +> Non-repeatable reads break the expectation that _data you read remains unchanged_ during a transaction. + +### Phantom Read + +A transaction re-runs a **range query** and sees **new or missing rows** that were inserted or deleted by another committed transaction. + +**Why it is dangerous:** +Even if individual rows don’t change, the **set of rows matching a condition** does. + +**Real-world example:** + +- You count how many users are above age 30. +- Another transaction inserts a new qualifying user. +- You run the same query again and get a different count. + +
+ +```mermaid +sequenceDiagram + participant T1 + participant T2 + + T1->>DB: SELECT COUNT(*) WHERE age > 30 + T2->>DB: INSERT age=35 + Commit + T1->>DB: SELECT COUNT(*) WHERE age > 30 (phantom) +``` + +
+ +**Key insight:** + +> Phantom reads show that isolation is not just about rows, it’s also about **ranges and predicates**. + +## Isolation Levels (SQL Standard) + +| Level | Dirty | Non-Repeatable | Phantom | +| ---------------- | --------- | -------------- | ---------------------- | +| Read Uncommitted | Allowed | Allowed | Allowed | +| Read Committed | Prevented | Allowed | Allowed | +| Repeatable Read | Prevented | Prevented | Allowed (SQL standard) | +| Serializable | Prevented | Prevented | Prevented | + +### How Isolation Is Implemented + +#### Lock-Based (Pessimistic) + +- Row locks +- Table locks +- Gap locks + +#### MVCC (Optimistic) + +- Snapshot reads +- Versioned rows + +
+ +```mermaid +flowchart TD + Tx[Transaction] + Snapshot[Snapshot at Start] + Versions[Multiple Row Versions] + Tx --> Snapshot --> Versions ``` -In each of these cases, PostgreSQL guarantees the level of isolation defined by the transaction's isolation level. For example, with `READ COMMITTED`, a transaction will see the latest committed data and avoid dirty reads, but non-repeatable reads are alloWed. +
## Durability -**Durability** ensures that once a transaction is committed, its effects are **permanent** and survive any system crashes, poOur failures, or other interruptions. +> After commit, data will survive crashes, power loss, and restarts. -- After a transaction commits, all changes made to the database are written to **persistent storage** (e.g., disk), ensuring that the data is not lost even if the system crashes shortly after the commit. -- Durability ensures that once a transaction has been committed, it cannot be undone (except by explicitly using a rollback mechanism in a new transaction). +### How Durability Is Achieved -**Example:** -If a transaction has been committed to transfer money from one account to another, the changes (debiting one account and crediting another) will persist even if the database system crashes right after the commit. +- Write-Ahead Logs flushed to disk +- fsync() +- Replication (in some systems) -```sql -BEGIN; -UPDATE accounts SET balance = balance - 100 WHERE account_id = 1; -- Debit -UPDATE accounts SET balance = balance + 100 WHERE account_id = 2; -- Credit -COMMIT; -- Once this commit happens, the changes are permanent --- A crash after this point won't undo the changes +
+ +```mermaid +sequenceDiagram + participant DB + participant Disk + + DB->>Disk: Flush WAL + Disk-->>DB: Acknowledged + DB->>App: Commit Success ``` -## Summary of ACID Properties +
+ +### Durability Trade-Offs + +| Strong Durability | Faster Performance | +| ----------------------- | ------------------------ | +| fsync per commit | Group commit | +| Synchronous replication | Asynchronous replication | + +Some databases allow relaxing durability requirements for improved speed. + +## ACID Across Major Databases + +### PostgreSQL + +- Full ACID +- MVCC-based isolation +- Serializable via SSI +- Strong durability by default + +### MySQL (InnoDB) + +- ACID compliant +- Repeatable Read default +- Gap locks prevent phantoms + +### Oracle + +- Very strong ACID +- Mature MVCC +- Excellent consistency guarantees + +### SQL Server + +- Lock-based + snapshot isolation +- Serializable supported + +### SQLite + +- ACID compliant +- Single-writer model +- WAL improves concurrency + +## ACID in Distributed Databases + +Maintaining ACID guarantees across multiple nodes is fundamentally more challenging than in a single-node system. + +### Challenges + +- **Network partitions**: We cannot reliably communicate between nodes at all times (CAP theorem) +- **Replication lag**: Data changes propagate asynchronously across replicas +- **Two-phase commit overhead**: Coordinating commits across multiple nodes is expensive and prone to failures +- **Consensus complexity**: Ensuring all nodes agree on a consistent state requires sophisticated consensus algorithms + +### Implementation Approaches + +#### Two-Phase Commit (2PC) + +A coordination protocol where a coordinator asks all participating nodes if they can commit, then either commits or aborts based on their responses. + +
+ +```mermaid +sequenceDiagram + participant Coordinator + participant Node1 + participant Node2 + + Coordinator->>Node1: Prepare (can you commit?) + Coordinator->>Node2: Prepare (can you commit?) + Node1-->>Coordinator: Yes/No + Node2-->>Coordinator: Yes/No + Coordinator->>Node1: Commit or Abort + Coordinator->>Node2: Commit or Abort +``` + +
+ +**Pros**: Strong consistency guarantees +**Cons**: Blocking operations, poor availability during partitions + +#### Eventual Consistency with Sagas + +Break distributed transactions into a series of local transactions coordinated by application logic. Each step can succeed or fail independently, with compensating transactions for rollbacks. + +#### Consensus-Based Systems (Raft, Paxos) + +Systems like CockroachDB use Raft consensus to replicate a consistent log across nodes, providing strong ACID guarantees even with network failures. This approach is more resilient than traditional 2PC. + +### Trade-Offs + +Distributed ACID is **expensive** because: + +- Coordination latency increases with each additional node +- Throughput decreases due to synchronous coordination +- Availability suffers during network partitions +- Complex failure scenarios require careful handling + +Many distributed systems (Cassandra, DynamoDB) explicitly choose BASE over distributed ACID for better scalability and availability. + +## BASE + +**BASE** is an acronym that stands for: + +- **B**asically Available - The system is available most of the time +- **S**oft state - The state of the system may change without explicit input (due to replication lag) +- **E**ventually consistent - The system will eventually reach a consistent state + +Used in **NoSQL & large-scale distributed systems** to prioritize availability and performance over immediate consistency. + +### BASE Philosophy + +> **Availability and scalability over strict immediate correctness** + +Rather than blocking operations to guarantee consistency, BASE systems accept temporary inconsistencies and resolve them asynchronously. This is a practical trade-off for systems operating at internet scale with thousands of nodes and users across the globe. + +### BASE vs ACID + +| Aspect | ACID | BASE | +| ------------------- | --------------------------------- | ------------------------------------ | +| Consistency model | Strong/immediate consistency | Eventual consistency | +| Transactions | Full ACID transactions | No / limited / local transactions | +| Availability | Can be reduced during failures | Designed to be always available | +| Latency | Higher due to coordination | Lower, optimized for speed | +| Partition tolerance | Limited (2PC fails on partitions) | High (designed to handle partitions) | +| Use case | Critical correctness (financial) | High-scale distributed systems | + +### How BASE Systems Work + +**Conflict Resolution Strategies:** + +- **Last-Write-Wins (LWW)**: The most recent write overwrites older writes (simple but can lose data) +- **Vector Clocks/Version Vectors**: Track causality to detect concurrent writes +- **Application-Level Resolution**: Let the application decide how to merge conflicting versions +- **CRDTs (Conflict-free Replicated Data Types)**: Data structures that guarantee convergence without coordination + +**Example: Adding an item to a set in a distributed system** + +- Node A adds item X +- Node B adds item Y +- Both additions are replicated asynchronously +- Eventually, all nodes converge to X, Y + +### Examples of BASE Systems -- **Atomicity:** Ensures that all operations in a transaction are treated as a single unit. If one operation fails, the entire transaction is rolled back. -- **Consistency:** Ensures that a transaction brings the database from one valid state to another, adhering to all rules and constraints. -- **Isolation:** Controls how operations in one transaction are isolated from others, preventing interference. Different isolation levels (Read Uncommitted, Read Committed, Repeatable Read, Serializable) provide different degrees of isolation. -- **Durability:** Guarantees that once a transaction is committed, the changes are permanent and will survive crashes. +- **DynamoDB** - Strongly consistent reads available but eventually consistent by default +- **Cassandra** - Tunable consistency; can configure reads/writes for performance vs. consistency +- **Riak** - Distributed key-value store with sophisticated conflict resolution +- **Couchbase** - NoSQL database with eventual consistency +- **Redis Cluster** - In-memory data structure store with asynchronous replication -## Importance of ACID +These systems: -ACID properties are essential for ensuring **data integrity** and **reliability** in database systems. They help maintain **consistent** data, handle **concurrent transactions** safely, and ensure that once data is committed, it is **permanent**. These properties are especially important in systems that require high reliability, such as **financial applications**, **e-commerce platforms**, and **inventory management systems**, where data accuracy and consistency are critical. +- Replicate aggressively across regions +- Accept temporary inconsistencies between replicas +- Resolve conflicts asynchronously using various strategies +- Prioritize low latency and high availability -## Eventual Consistency +## When to Use ACID vs BASE -**Eventual Consistency** is a consistency model used in distributed databases and systems where it is acceptable for data to be temporarily inconsistent, but the system guarantees that, given enough time, all nodes in the system will eventually converge to a consistent state. This model is primarily associated with **NoSQL** databases and distributed systems that prioritize availability and partition tolerance over strict consistency, as defined by the **CAP theorem**. +### Use ACID When -In an **eventually consistent** system, updates to the data are made available to all replicas, but they may not be immediately reflected across all replicas. The system will eventually propagate these changes to all nodes, ensuring that, over time, every replica will have the same data. +- **Financial transactions**: Money transfers, payments, accounting +- **Critical data correctness**: Medical records, legal contracts +- **Business invariants must never break**: Unique constraints, referential integrity +- **Small to medium scale**: Single-region or limited geographic distribution +- **Lower throughput requirements**: Where latency is less critical than correctness -### Key Characteristics of Eventual Consistency +### Use BASE When -1. **Temporary Inconsistency**: - At any point in time, different replicas of the data might have slightly different versions of the data. This can lead to situations where a read operation on different nodes may return different results, but eventually, all replicas will become consistent. +- **Massive global scale**: Millions of users across multiple continents +- **Low latency is critical**: User-facing applications require response times < 100ms +- **Temporary inconsistency is acceptable**: Social media feeds, inventory counts (approximate is fine) +- **High availability**: System must remain operational during network partitions +- **Write-heavy workloads**: Need to scale writes horizontally -2. **Eventual Convergence**: - Over time, the system ensures that all updates are propagated across all nodes. Eventually, all replicas will become consistent, hence the term "eventual consistency." +### Hybrid Approaches -3. **High Availability**: - Eventual consistency allows the system to prioritize availability and continue operating even when some parts of the system are temporarily unavailable or partitioned. This is particularly useful for systems with high availability requirements, like global-scale Web applications. +Many modern systems use a **polyglot persistence** strategy: -4. **Concurrent Writes**: - Eventual consistency allows for concurrent updates to the same data from different locations or users. Conflicts might arise, but conflict resolution mechanisms (such as last-write-wins or custom conflict resolution logic) ensure that the system can converge to a consistent state. +- **ACID database** for critical data (user accounts, transactions) +- **BASE system** for high-volume, eventually-consistent data (activity feeds, recommendations) +- **Cache layer** for read-heavy data