|
| 1 | +--- |
| 2 | +icon: material/connection |
| 3 | +--- |
| 4 | + |
| 5 | +# Connection recovery |
| 6 | + |
| 7 | +PostgreSQL database connections are expensive to create so PgDog does its best not to close them unless absolutely necessary. In case a client disconnects before fully processing a query response, PgDog will attempt to preserve the connection using several recovery steps. |
| 8 | + |
| 9 | +## Abandoned transactions |
| 10 | + |
| 11 | +If a client disconnects abruptly while inside a transaction, the transaction is considered abandoned and PgDog will automatically execute a `ROLLBACK`, making sure none of its changes are persisted in the database. |
| 12 | + |
| 13 | +This is a common occurrence if there is a bug that causes the application to crash while executing multiple statements inside a manually started transaction, for example: |
| 14 | + |
| 15 | +=== "Rails" |
| 16 | + ```ruby |
| 17 | + ActiveRecord.transaction do |
| 18 | + user = User.find(5) |
| 19 | + # crash happens here. |
| 20 | + end |
| 21 | + ``` |
| 22 | +=== "SQLAlchemy" |
| 23 | + ```python |
| 24 | + with session.begin(): |
| 25 | + user = session.get(User, 5) |
| 26 | + # crash happens here. |
| 27 | + ``` |
| 28 | +=== "Go" |
| 29 | + ```go |
| 30 | + tx, _ := db.Begin() |
| 31 | + row := tx.QueryRow("SELECT * FROM users WHERE id = $1", 5) |
| 32 | + // crash happens here. |
| 33 | + ``` |
| 34 | + |
| 35 | +### Connection storms |
| 36 | + |
| 37 | +By preserving connections, PgDog protects the database against connection storms. Other connection poolers like PgBouncer close server connections without attempting any recovery. |
| 38 | + |
| 39 | +When the application restarts, the pooler must recreate all of these connections at once, causing thousands of server connections to be opened and closed in rapid succession. This leads to unnecessary contention on database resources and can cause 100% CPU spikes on the database. |
| 40 | + |
| 41 | +## Abandoned queries |
| 42 | + |
| 43 | +A client can abruptly disconnect while receiving query response data from the server. This can happen due to out-of-memory errors or hardware failure, for example: |
| 44 | + |
| 45 | +=== "Rails" |
| 46 | + ```ruby |
| 47 | + orders = Order.where(user_id: 5) |
| 48 | + # ^ crash happens inside `pg`, |
| 49 | + # while receiving multiple rows |
| 50 | + ``` |
| 51 | +=== "SQLAlchemy" |
| 52 | + ```python |
| 53 | + orders = session.execute( |
| 54 | + select(Order).where(Order.user_id == 5) |
| 55 | + ).all() |
| 56 | + # ^ crash happens while receiving multiple rows |
| 57 | + ``` |
| 58 | +=== "Go" |
| 59 | + ```go |
| 60 | + rows, _ := db.Query("SELECT * FROM orders WHERE user_id = $1", 5) |
| 61 | + for rows.Next() { |
| 62 | + // crash happens here while iterating over rows |
| 63 | + } |
| 64 | + ``` |
| 65 | + |
| 66 | +PgDog will detect this and drain server connections, restoring them to their normal state, before returning them back to the connection pool. The drain mechanism works by receiving and discarding `DataRow` messages and sending [`Sync`](https://www.postgresql.org/docs/current/protocol-message-formats.html#PROTOCOL-MESSAGE-FORMATS-SYNC) to the server to resynchronize the extended protocol state. |
| 67 | + |
| 68 | +Just like [abandoned transactions](#abandoned-transactions), this protects PostgreSQL databases from connection storms caused by unreliable clients. If the client was executing a transaction, it will be rolled back as well. |
| 69 | + |
| 70 | +### Configuration |
| 71 | + |
| 72 | +Connection recovery is an optional feature, enabled by default. You can change how it behaves through configuration: |
| 73 | + |
| 74 | +```toml |
| 75 | +[general] |
| 76 | +connection_recovery = "recover" |
| 77 | +``` |
| 78 | + |
| 79 | +| Configuration value | Description | |
| 80 | +|-|-| |
| 81 | +| `recover` | Attempt full connection recovery, including rollback and resynchronization. This is the default. | |
| 82 | +| `rollback_only` | Rollback abandoned transactions but drop the connection if a query was abandoned mid-response. | |
| 83 | +| `drop` | Disable connection recovery and close the server connection (identical to PgBouncer). | |
| 84 | + |
| 85 | +To make sure abandoned server connections don't block normal operations, PgDog supports a configurable timeout on the recovery operation. If connection recovery doesn't complete in time, the connection will be closed: |
| 86 | + |
| 87 | +```toml |
| 88 | +[general] |
| 89 | +rollback_timeout = 5_000 |
| 90 | +``` |
| 91 | + |
| 92 | +## Client connections |
| 93 | + |
| 94 | +Just like server connections, PgDog can maintain client connections (application --> PgDog) during incidents. This helps preserve application-side connection pools and avoids re-creating thousands of connections unnecessarily. |
| 95 | + |
| 96 | +While enabled by default, some applications don't behave well when their queries return errors instead of results. Therefore, this feature is configurable and can be disabled: |
| 97 | + |
| 98 | +```toml |
| 99 | +[general] |
| 100 | +client_connection_recovery = "drop" |
| 101 | +``` |
| 102 | + |
| 103 | +| Configuration value | Description | |
| 104 | +|-|-| |
| 105 | +| `recover` | Attempt to maintain client connections open after database-related errors, like `checkout timeout`. | |
| 106 | +| `drop` | Disable connection recovery and close the client connection (identical to PgBouncer). | |
0 commit comments