Database Reliability Engineering: 6 Proven Steps MinervaDB Runs on Every Engine

Database reliability engineering is how we run data platforms at MinervaDB: service level objectives instead of gut feel, error budgets instead of endless change freezes, and automation instead of heroics. We apply the same discipline to PostgreSQL, MySQL, MariaDB, SQL Server, Oracle, Db2, MongoDB, ClickHouse and Redis, and to the managed services our clients run on: Amazon RDS, Amazon Aurora, Google Cloud SQL, AlloyDB, Google BigQuery and Azure SQL.

This post explains how we deliver database reliability engineering in practice: how we choose objectives, where we read the truth on each engine, how alerts page people, how incidents close, and how backups, schema changes and failovers are proven rather than assumed. Where a practice is best shown in code, we include the configuration or SQL we use.

What database reliability engineering means at MinervaDB

Traditional database administration asks whether the server is up. Database reliability engineering asks whether the service is meeting the promise the business depends on, and how much room is left before it does not. The unit of work is not a server but a service: the orders database, the ledger, the analytics warehouse behind executive dashboards.

Four ideas from site reliability engineering do most of the work. Service level indicators measure what users experience. Service level objectives set the target. The error budget is the gap between perfect and the target, and it decides when reliability work outranks new features. Toil, the manual and repetitive work that scales with load, is treated as a defect to be engineered away.

Step one: service level objectives that mean something

An objective is only useful if someone would notice when it is missed. For a transactional database we usually define availability as the share of successful probe transactions, latency as a percentile of real query time for named critical statements, and durability as the achieved recovery point in restore drills. Analytical platforms add freshness: how far behind the source the data is allowed to fall.

We measure availability with a synthetic prober that performs a real read and a real write against each service, so a database that is up but refusing writes still counts as down. The rules below turn that probe into multi-window burn-rate alerts, the approach described in the Google SRE workbook. A fast burn pages the on-call engineer; a slow burn opens a ticket.

# Prometheus 2.x rules: 99.9% availability SLO for one database service.
# db_probe_* counters come from our synthetic prober, which runs a real
# read and write transaction against the service every 10 seconds.
groups:
  - name: slo_orders_db
    rules:
      - record: slo:orders_db_probe_errors:ratio_rate5m
        expr: 1 - (sum(rate(db_probe_success_total{service="orders-db"}[5m]))
                 / sum(rate(db_probe_attempts_total{service="orders-db"}[5m])))
      - record: slo:orders_db_probe_errors:ratio_rate30m
        expr: 1 - (sum(rate(db_probe_success_total{service="orders-db"}[30m]))
                 / sum(rate(db_probe_attempts_total{service="orders-db"}[30m])))
      - record: slo:orders_db_probe_errors:ratio_rate1h
        expr: 1 - (sum(rate(db_probe_success_total{service="orders-db"}[1h]))
                 / sum(rate(db_probe_attempts_total{service="orders-db"}[1h])))
      - record: slo:orders_db_probe_errors:ratio_rate6h
        expr: 1 - (sum(rate(db_probe_success_total{service="orders-db"}[6h]))
                 / sum(rate(db_probe_attempts_total{service="orders-db"}[6h])))

      # Fast burn: 2% of a 30-day budget gone in one hour -> page now
      - alert: OrdersDbErrorBudgetFastBurn
        expr: slo:orders_db_probe_errors:ratio_rate1h > (14.4 * 0.001)
          and slo:orders_db_probe_errors:ratio_rate5m > (14.4 * 0.001)
        labels:
          severity: S1
        annotations:
          runbook: https://runbooks.example.internal/orders-db/availability

      # Slow burn: 5% of the budget gone in six hours -> ticket, same business day
      - alert: OrdersDbErrorBudgetSlowBurn
        expr: slo:orders_db_probe_errors:ratio_rate6h > (6 * 0.001)
          and slo:orders_db_probe_errors:ratio_rate30m > (6 * 0.001)
        labels:
          severity: S2

Burn-rate alerting is the core of how database reliability engineering avoids alert fatigue. A one-minute CPU spike does not page anyone; a sustained error rate that would exhaust a month of budget in two days does.

Step two: instrument every engine at the source

Dashboards built only from host metrics show symptoms, not causes, which is why database reliability engineering is anchored in the engine itself. Database reliability engineering starts from the views each engine exposes about itself: where time goes, what is waiting, how far replicas are behind and how close the system is to a hard limit such as transaction ID wraparound or a full oplog window.

Two details matter in practice. On Oracle, AWR and ASH require the Diagnostics Pack licence, so on unlicensed estates we use Statspack and V$ views instead. On Amazon RDS and Aurora, CloudWatch Database Insights is now the primary performance surface, and we pair it with engine views queried directly, because provider dashboards aggregate away the detail an incident needs.

Step three: respond within the severity targets

Every client engagement carries the same response targets: Severity 1 in 15 minutes around the clock, Severity 2 in 12 hours, Severity 3 in 24 hours and Severity 4 in 48 hours. A named engineer takes incident command, one person speaks to the client, and the first minutes go to collecting evidence before anything is changed.

Evidence disappears quickly: the blocking session ends, the long transaction commits, the replica catches up. So the alert itself triggers a read-only capture of the state that matters. This PostgreSQL example records active sessions, blocking chains and replication state into a timestamped folder.

#!/usr/bin/env bash
# Read-only evidence capture, run automatically when a PostgreSQL alert fires.
# Credentials come from ~/.pgpass or ${PGPASSWORD}; nothing secret lives in this file.
set -euo pipefail
OUT="/var/lib/dre/evidence/$(date -u +%Y%m%dT%H%M%SZ)-${PGHOST}"
mkdir -p "${OUT}"
PSQL="psql -X -A -F, -h ${PGHOST} -U ${PG_USER} -d ${PG_DATABASE} -v ON_ERROR_STOP=1"

${PSQL} -c "SELECT now(), pg_is_in_recovery(), version();" > "${OUT}/server.csv"

${PSQL} -c "SELECT pid, usename, state, wait_event_type, wait_event,
                   now() - xact_start  AS xact_age,
                   now() - query_start AS query_age,
                   LEFT(query, 200)    AS query_head
            FROM pg_stat_activity
            WHERE state <> 'idle'
            ORDER BY xact_start NULLS LAST;" > "${OUT}/activity.csv"

${PSQL} -c "SELECT blocked.pid  AS blocked_pid,
                   blocking.pid AS blocking_pid,
                   LEFT(blocking.query, 200) AS blocking_query
            FROM pg_stat_activity AS blocked
            JOIN LATERAL unnest(pg_blocking_pids(blocked.pid)) AS b(pid) ON true
            JOIN pg_stat_activity AS blocking ON blocking.pid = b.pid;" > "${OUT}/blocking.csv"

${PSQL} -c "SELECT application_name, state, sync_state, replay_lag
            FROM pg_stat_replication;" > "${OUT}/replication.csv"

echo "evidence written to ${OUT}"

Equivalent captures exist for each engine: performance_schema and InnoDB status for MySQL and MariaDB, DMV snapshots for SQL Server, ASH or V$ views for Oracle, MON_GET functions for Db2, $currentOp for MongoDB and system tables for ClickHouse. In database reliability engineering, the capture runs first and the fix second.

Step four: learn from every incident

In our database reliability engineering practice, every Severity 1 and Severity 2 incident closes with a written, blameless root cause analysis: a timeline, the trigger, the contributing conditions, why detection took as long as it did, and actions with owners and dates. Actions are tracked to closure in the same way as feature work, and the error budget shows whether they are paying off.

Patterns across incidents matter more than any single one. If three incidents in a quarter trace back to unbounded connection growth, the action is not a fourth runbook but pooling, timeouts and a capacity alert that fires weeks earlier.

Step five: make change safe

Many production incidents begin with a change: a deployment, a migration, a parameter edit. Database reliability engineering does not slow change down; it makes each change small, observable and reversible. For schema changes on PostgreSQL that means bounded lock waits, constraints added without a full-table lock, and indexes built concurrently.

-- PostgreSQL 12+: a schema change that cannot stall production behind a lock queue
SET lock_timeout      = '3s';     -- give up rather than block every writer behind us
SET statement_timeout = '15min';

ALTER TABLE orders
    ADD COLUMN fulfilment_region TEXT;            -- metadata-only, no table rewrite

ALTER TABLE orders
    ADD CONSTRAINT ck_orders_fulfilment_region
    CHECK (fulfilment_region IN ('APAC', 'EMEA', 'AMER')) NOT VALID;   -- new rows only

-- Validates existing rows without blocking reads or writes
ALTER TABLE orders VALIDATE CONSTRAINT ck_orders_fulfilment_region;

-- Built without blocking writes (cannot run inside a transaction block)
CREATE INDEX CONCURRENTLY idx_orders_fulfilment_region
    ON orders (fulfilment_region, created_at);

-- Verify: the index is valid and the constraint is enforced
SELECT indexrelid::regclass AS index_name, indisvalid
FROM pg_index
WHERE indexrelid = 'idx_orders_fulfilment_region'::regclass;

SELECT conname, convalidated
FROM pg_constraint
WHERE conname = 'ck_orders_fulfilment_region';

The same principles apply elsewhere: online DDL or gh-ost for MySQL, resumable index operations for SQL Server, rolling index builds in MongoDB, and ON CLUSTER changes in ClickHouse rolled out shard by shard. Every change has a verification query before and a validation query after.

Step six: prove recovery with drills

A backup that has never been restored is a hypothesis. We schedule restore drills, failover drills and region evacuations, measure the recovery time and recovery point actually achieved, and report the gap against the objective. Drills run in pre-production first and in production only where the safety nets are in place.

The restore drill below shows the guard rails we put on anything that writes to disk: an emptiness check, an explicit typed confirmation, a verification before and integrity checks after. It records the achieved recovery time as a number, not an impression.

#!/usr/bin/env bash
# Quarterly restore drill, pgBackRest 2.x: restore the latest backup to an ISOLATED host
# and record the achieved RTO. Never run this against a production data directory.
set -euo pipefail
STANZA="${PG_STANZA}"
TARGET="/srv/drill/pgdata"

# Confirmation gate: the target must be empty and not belong to a running cluster
if [ -n "$(ls -A "${TARGET}" 2>/dev/null)" ]; then
  echo "ABORT: ${TARGET} is not empty"; exit 1
fi
read -r -p "Restore stanza ${STANZA} into ${TARGET} on $(hostname)? Type RESTORE to continue: " ok
[ "${ok}" = "RESTORE" ] || { echo "aborted"; exit 1; }

pgbackrest --stanza="${STANZA}" info                     # verification before
start=$(date +%s)
pgbackrest --stanza="${STANZA}" --pg1-path="${TARGET}" --type=default restore
pg_ctl -D "${TARGET}" -o "-p 6543" -w start
end=$(date +%s)
echo "achieved RTO: $(( end - start )) seconds"

# Validation after: the restored cluster answers and its data is internally consistent
psql -p 6543 -d "${PG_DATABASE}" -c "SELECT pg_last_xact_replay_timestamp() AS recovered_to;"
psql -p 6543 -d "${PG_DATABASE}" -c "CREATE EXTENSION IF NOT EXISTS amcheck;"
psql -p 6543 -d "${PG_DATABASE}" -c "SELECT bt_index_check(c.oid)
    FROM pg_class AS c JOIN pg_am AS am ON am.oid = c.relam
    WHERE am.amname = 'btree' AND c.relkind = 'i' LIMIT 200;"

Database reliability engineering on Amazon RDS, Aurora, Cloud SQL, AlloyDB and BigQuery

Managed services remove host work, not reliability engineering. The provider operates hardware, storage durability, minor version patching and the failover mechanism. Service level objectives, alerting, capacity, cost, query and schema health, and proof that backups restore remain the customer's responsibility, and in our engagements they become ours.

On Amazon RDS and Aurora we time failovers end to end, including application reconnects, and keep backup copies in a separate account. On Cloud SQL and AlloyDB we plan maintenance windows, watch Query Insights for plan regressions and test cross-region replicas. On BigQuery, reliability is about query latency and cost predictability, so we define objectives directly from job metadata.

-- BigQuery: a daily latency SLI for an analytics workload, from INFORMATION_SCHEMA.JOBS
SELECT DATE(creation_time)                                              AS run_date,
       COUNT(*)                                                         AS queries,
       APPROX_QUANTILES(TIMESTAMP_DIFF(end_time, start_time, MILLISECOND), 100)[OFFSET(95)]
                                                                        AS p95_ms,
       COUNTIF(TIMESTAMP_DIFF(end_time, start_time, SECOND) <= 10) / COUNT(*)
                                                                        AS share_within_10s,
       ROUND(SUM(total_slot_ms) / 1000 / 3600, 1)                       AS slot_hours
FROM `region-us`.INFORMATION_SCHEMA.JOBS
WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 28 DAY)
  AND job_type = 'QUERY'
  AND state    = 'DONE'
  AND error_result IS NULL
  AND EXISTS (SELECT 1 FROM UNNEST(labels) AS l WHERE l.key = 'workload' AND l.value = 'exec_dashboards')
GROUP BY run_date
ORDER BY run_date DESC;

That query gives a daily p95 and the share of dashboard queries finishing within ten seconds, which becomes the latency SLI for an analytics service. Slot hours alongside it show whether meeting the objective is costing more each week.

Capacity, toil and the engineering backlog

Reliability problems that arrive with growth are predictable, and database reliability engineering treats capacity as an objective like any other. We forecast CPU, memory, storage, IOPS and connection headroom from measured trends, and raise capacity actions while there is still room to plan them. On managed services the same forecast covers instance classes, storage autoscaling limits, slot reservations and the cost of each step.

Toil is tracked as a number: hours per month of manual, repetitive operational work. When it grows, it becomes engineering work: automated failover checks, self-service read replicas, scripted user provisioning, alert auto-remediation for well-understood cases. Database reliability engineering exists to keep that number falling while the estate grows.

How a database reliability engineering engagement starts

Most engagements begin with a two-to-four-week baseline. We inventory every engine, version and managed service in the estate, map each one to the business services that depend on it, and record the current state of monitoring, backups, high availability and on-call. The output is a ranked list of reliability risks, each tied to the metric or configuration that shows it.

From that baseline we agree a first set of service level objectives with the people who own the business outcome, not only with the database team. Database reliability engineering only works when an objective has an owner who cares when it is missed.

The first quarter usually closes the largest gaps: an untested backup chain, a failover nobody has timed, alerts that page on noise. After that, database reliability engineering settles into a steady rhythm of monthly reviews, quarterly drills and an engineering backlog driven by the error budget.

Why teams choose MinervaDB for database reliability engineering

More than 900 enterprises work with MinervaDB. Clients cite the same reasons for choosing our database reliability engineering practice:

• One practice across every engine and cloud. PostgreSQL, MySQL, MariaDB, SQL Server, Oracle, Db2, MongoDB, ClickHouse, Redis and Valkey, plus Amazon RDS, Aurora, Cloud SQL, AlloyDB, BigQuery, Azure SQL, Snowflake and Databricks.
• Measured, not asserted. Every recommendation cites the metric or system view behind it, and every change has a verification before and after.
• Senior engineers. Principal-level engineers backed by 200+ years of combined leadership experience and 15+ years of company depth, delivering from 46 cities.
• Response targets that hold. Severity 1 in 15 minutes, Severity 2 in 12 hours, Severity 3 in 24 hours, Severity 4 in 48 hours, on every engine.
• Vendor neutrality. We sell no licences, so we will say when a managed service or engine is the wrong fit.

Database reliability engineering is usually delivered alongside our remote DBA services and full-stack database infrastructure practice, and complements an internal team rather than replacing it.

Frequently asked questions

What is database reliability engineering?

It applies site reliability engineering to data platforms: service level objectives, error budgets, engine-level observability, structured incident response, blameless root cause analysis, safe change and rehearsed recovery.

How is database reliability engineering different from a remote DBA service?

A remote DBA service keeps databases running and handles routine work. Database reliability engineering adds measurable objectives, error budgets and an engineering backlog that removes recurring incidents and toil. We usually deliver both together.

Does database reliability engineering apply to Amazon RDS, Aurora, Cloud SQL and BigQuery?

Yes. The provider operates the infrastructure, but objectives, alerting, capacity, cost, schema and query health, and proven recovery remain the customer's responsibility, and that is where we work.

What response times does MinervaDB commit to?

Severity 1 in 15 minutes, 24×7×365; Severity 2 in 12 hours; Severity 3 in 24 hours; Severity 4 in 48 hours.

The configurations and scripts in this post are illustrative and version-pinned. Test every change in a non-production environment first, verify backups by restoring them, and maintain a robust disaster-recovery posture before applying anything to production.

Ready to put database reliability engineering behind your data platform? Talk to a MinervaDB principal architect, or email contact@minervadb.com.