Troubleshooting PostgreSQL Streaming Replication Lag in Production

PostgreSQL replication lag is only a useful phrase once you say which lag you mean. A standby can be receiving WAL at full line rate yet replaying it minutes behind; it can be replaying instantly but starved by a saturated network; on synchronous setups, "lag" surfaces on the primary as commit latency instead of on the standby at all. Effective PostgreSQL replication lag troubleshooting therefore starts by decomposing the pipeline — generate → send → receive/flush → replay — measuring each stage, and only then fixing the stage that is actually behind. This is the sequence our 24×7 team applies on streaming-replication fleets from PostgreSQL 13 through 17.

Measuring PostgreSQL Replication Lag: The Four-Stage Pipeline

On the primary, pg_stat_replication exposes per-standby LSN positions and, since PostgreSQL 10, direct time-based lag columns:

SELECT
    application_name,
    client_addr,
    state,
    sync_state,
    pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), sent_lsn))   AS send_lag_bytes,
    pg_size_pretty(pg_wal_lsn_diff(sent_lsn, flush_lsn))              AS flush_lag_bytes,
    pg_size_pretty(pg_wal_lsn_diff(flush_lsn, replay_lsn))            AS replay_lag_bytes,
    write_lag,
    flush_lag,
    replay_lag
FROM pg_stat_replication;

The decomposition tells you where to look. send lag growing: the primary can't push WAL fast enough — network bandwidth, or a WAL-generation burst. flush lag growing: the standby receives but can't persist — standby WAL-disk latency. replay lag growing while the others stay flat: the single-threaded startup process on the standby can't apply as fast as WAL arrives, or replay is deliberately paused by a conflict/delay — this is the most common production case and has its own section below. On the standby itself, corroborate with:

SELECT
    NOW() - pg_last_xact_replay_timestamp()  AS replay_delay,
    pg_last_wal_receive_lsn(),
    pg_last_wal_replay_lsn();

One reading caveat: replay_delay only advances when transactions replay, so on an idle primary it grows spuriously; the LSN-difference view is the trustworthy signal. Alert on both bytes and seconds — bytes for capacity trending, seconds for the RPO/staleness promise you've made to applications.

Stage 1 Bottlenecks: WAL Generation Bursts on the Primary

Many "replication is lagging" pages are really "the primary generated a WAL tsunami." Quantify generation rate before blaming the transport:

-- PostgreSQL 14+: aggregate WAL statistics
SELECT wal_records, wal_fpi, wal_bytes,
       stats_reset
FROM pg_stat_wal;

Trend wal_bytes per minute across the incident. The classic burst sources: bulk loads and mass UPDATEs, index builds, VACUUM FULL/CLUSTER, and full-page-image storms right after checkpoints (high wal_fpi share — spreading checkpoints via a larger max_wal_size reduces FPI pressure). Fixes are demand-side: batch large writes into smaller transactions spread over time, schedule maintenance off-peak, and on 15+ consider wal_compression = zstd (reload) — it trades primary CPU for materially less WAL to ship, which helps every downstream stage. Verify effect by re-measuring wal_bytes/min and watching send lag flatten.

Stage 2–3 Bottlenecks: Network and Standby Storage

Sustained send lag with flat generation rate means transport. Measure the path honestly — cross-AZ and cross-region links have both bandwidth caps and latency floors; a single walsender TCP stream over a 60 ms RTT path will not saturate the link without window tuning. Practical levers: OS-level TCP buffer sizing on both ends, ensuring no unintended traffic shaping, and for chronically constrained WAN links, evaluating whether the standby should instead ship via a nearer relay (cascading replication: a same-region standby feeds the remote one, cutting the primary's send load and the WAN retransmit blast radius).

Flush lag isolates standby WAL-device latency: confirm with iostat on the standby's pg_wal volume. The fix list is unexciting and effective: dedicated low-latency volume for pg_wal, correct write-cache configuration, and on cloud volumes, provisioned IOPS that match the primary's — an economy-class standby behind a performance-class primary is a designed-in PostgreSQL replication lag machine that also silently caps your failover capacity.

Stage 4: PostgreSQL Replication Lag in Replay — The Single-Threaded Truth

WAL replay on a standby is applied by one startup process. A primary with 64 cores of concurrent write throughput can outrun one standby core, and no parameter makes redo multi-threaded in core PostgreSQL through 17. What you can fix:

Replay-blocking conflicts. With hot_standby_feedback = off, replay must wait (up to max_standby_streaming_delay, default 30 s) or cancel standby queries that pin old row versions. Check the score:

SELECT * FROM pg_stat_database_conflicts
WHERE datname = current_database();

Decide the trade explicitly per standby role: analytics standbys → hot_standby_feedback = on (accepting primary-side bloat from held horizons) or a longer standby delay (accepting lag); HA standbys → keep feedback off, short delays, and no long queries allowed. Mixed-use standbys are the anti-pattern; split the roles.

Replay I/O starvation. Redo is read-modify-write against data files; a cold standby buffer cache turns replay into random-read torture. On 15+, recovery_prefetch = on (default try) with a tuned maintenance_io_concurrency materially improves redo throughput on capable storage — verify via pg_stat_recovery_prefetch. Ensure the standby's data volume, not just its WAL volume, matches primary-class IOPS.

Deliberate delay. Confirm nobody set recovery_min_apply_delay on the standby you're paging about — a delayed standby intended for oops-recovery duty that ended up in the read pool produces exactly this ticket.

Synchronous Replication: PostgreSQL Replication Lag Wearing a Different Mask

Under synchronous_commit = on with synchronous_standby_names set, standby slowness doesn't accumulate as standby lag — it becomes primary commit latency, visible as SyncRep wait events in pg_stat_activity. Triage identically (which stage of which standby is slow), but manage the policy dimension too: quorum-based ANY 1 (s1, s2) configurations tolerate one slow standby; FIRST-based lists serialize on the named order. Downgrading durability (synchronous_commit = remote_write or local) is an RPO decision for the business, not a tuning knob — document it as such, per role if adopted.

Slots, Retention, and the Disk-Full Failure Mode

Replication slots guarantee WAL retention for a disconnected or lagging standby — which means a dead standby with a live slot will eventually fill the primary's pg_wal and take the primary down. Monitor retained WAL per slot and cap it:

SELECT slot_name, active,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal,
       wal_status,
       safe_wal_size
FROM pg_replication_slots;

Set max_slot_wal_keep_size (PostgreSQL 13+, reload; e.g., 200 GB sized to your standby-rebuild tolerance) so a runaway slot is invalidated (wal_status = 'lost') before the primary's disk fills. The trade is explicit: an invalidated slot means that standby must be rebuilt (pg_basebackup or restored from backup + WAL archive) — an operational cost you accept to protect primary availability. Alert on wal_status transitions and on inactive slots older than your policy window; a standing "orphaned slot" check belongs in every PostgreSQL runbook.

Archive Interplay: When Catch-Up Runs Through restore_command

Streaming is only one of a standby's two WAL sources. When a standby falls behind past the primary's retained WAL — slot invalidated, or no slot and wal_keep_size exceeded — it silently fails over to archive recovery, fetching segments via restore_command until it is close enough to resume streaming. This mode change is where several distinct PostgreSQL replication lag pathologies hide, and they are diagnosed from the standby's log (restored log file "000000010000A4C300000041" from archive lines alternating with streaming attempts) rather than from pg_stat_replication, which shows nothing while the standby isn't streaming.

Archive catch-up throughput is gated by per-segment fetch latency: a restore_command that takes 400 ms per 16 MB segment caps recovery at ~40 MB/s regardless of storage speed, and object-store backends (pgBackRest, WAL-G) reach acceptable rates only with parallel/async prefetch enabled — pgBackRest's archive-async=y with tuned process-max, WAL-G's prefetch settings — which routinely improves catch-up by integer factors. Watch the primary side of the same coin: a failing or slow archive_command backs up pg_wal on the primary exactly like a dead slot does; monitor pg_stat_archiver.failed_count and the age of last_archived_wal with the same severity as slot retention, and consider archive_timeout's trade (bounded archive staleness versus a steady stream of padded segments on quiet systems).

Also size wal_segment_size awareness into WAN math — clusters initialized at 64 MB segments (an initdb-time choice) fetch fewer, larger objects, which materially changes object-store archive economics. Finally, rehearse the mode transitions: a standby must demonstrably move archive → streaming (log line confirms) as it catches up, and a standby oscillating between the two is telling you its streaming source can't hold it — pointing you back to the four-stage decomposition with which this guide began.

Verification After Any PostgreSQL Replication Lag Fix

Close the loop with the same instrumentation across one representative peak: per-stage lag columns flat and near zero at steady state; replay_lag spikes correlated only with known maintenance; pg_stat_database_conflicts counters static; slot retained_wal bounded; and on synchronous setups, SyncRep wait share in your wait-event sampling back to baseline. Then rehearse the thing PostgreSQL replication lag actually threatens: run your failover drill and measure real RPO/RTO against the promises — PostgreSQL replication lag monitoring exists to defend those numbers, not as an end in itself.

PostgreSQL Replication Lag Inside an HA Stack: Patroni’s View of the World

When streaming replication lives under Patroni, PostgreSQL replication lag stops being merely a performance metric and becomes failover eligibility. Patroni's maximum_lag_on_failover (bytes, default 1 MB order) disqualifies any replica whose lag exceeds the threshold from being promoted — so a chronically lagging standby silently converts your three-node HA cluster into a one-candidate cluster, and you discover it during the outage. Audit the live view the same way the failover logic does: patronictl list reports per-member lag, and the REST endpoints (GET /replica?lag=16MB style health checks) are what your load balancer should already be using to route read traffic away from stale replicas. Three Patroni-specific rules from our HA practice.

First, alert on the margin between observed replica lag and maximum_lag_on_failover, not just on absolute lag — the actionable event is "one promotion candidate left," which arrives earlier than any user-visible staleness. Second, tune ttl, loop_wait, and lag thresholds together: aggressive failover timing plus generous lag tolerance can promote a replica that is behind, converting a node failure into data loss within your own RPO math. Third, after any switchover, verify timeline convergence — pg_stat_replication on the new leader must show all members streaming on the new timeline, and any member stuck on the old one (visible via pg_rewind requirements in the Patroni log) is your next PostgreSQL replication lag incident already in progress.

Designing PostgreSQL Replication Lag Monitoring That Predicts Instead of Reports

A single PostgreSQL replication lag gauge tells you that you were behind; a monitoring design tells you why and what happens next. The exporter layer (postgres_exporter or pgwatch) should ship, per standby: the three stage-lag byte deltas and the two time-lag columns from pg_stat_replication; standby-side pg_last_xact_replay_timestamp() delay; slot retained_wal and wal_status; and primary pg_stat_wal.wal_bytes rate. From those, build three derived signals that do the actual predicting. Catch-up ratio: replay rate ÷ WAL generation rate over a 5-minute window — a standby at ratio 0.85 during peak is mathematically guaranteed to fall behind until traffic drops; alert on ratio < 1 sustained, which fires long before any absolute threshold.

Time-to-slot-invalidation: (max_slot_wal_keep_size − retained_wal) ÷ generation rate — the countdown until a lagging standby needs a rebuild; page when it drops under your rebuild duration. Drain time: current lag bytes ÷ (replay rate − generation rate) — the honest ETA you give stakeholders, which is infinite whenever the catch-up ratio is under 1, a fact the gauge alone hides. Route these with the same severity as disk-space alerts: replication debt is a durability and availability signal — it is your live RPO, measured.

Cascading Topologies: Spending PostgreSQL Replication Lag to Buy Capacity

Cascading replication — standbys feeding standbys — is the standard answer to two scaling problems: a primary whose walsender count and network egress are saturating (each direct standby costs primary CPU and bandwidth), and geographically distributed read fleets where shipping WAL across the WAN once, then fanning out locally, beats N parallel transoceanic streams. The cost is arithmetic: a second-tier standby's lag is the sum of both hops, and a relay failure orphans its entire subtree until you re-point primary_conninfo (a config change plus restart of the walreceiver on each downstream node — scriptable, but rehearse it).

Design rules we apply: keep HA promotion candidates on tier one, directly attached to the primary, so failover math never includes relay hops; give the relay node primary-class WAL storage since it both replays and re-serves; use slots at every hop with max_slot_wal_keep_size capped at each level, because a slow leaf can otherwise fill its relay's disk exactly as it would a primary's; and monitor per-hop stage lag, since the decomposition discipline from earlier in this guide applies at every link — a tier-two PostgreSQL replication lag alert with a healthy tier one indicts the relay's send path, not the primary.

On PostgreSQL 14+, remember relays can serve reads too; a well-placed regional relay is often the cheapest read-scaling capacity you own, provided its dual role is reflected in its resource class.

Quick Reference: PostgreSQL Replication Lag Triage

Frequently Asked Questions About PostgreSQL Replication Lag

What PostgreSQL replication lag is “acceptable”?

Whatever your stated RPO and read-staleness contracts say — engineering targets follow the business promise. Sub-second PostgreSQL replication lag is routine for same-region streaming on matched hardware; define alerts as a fraction of the contract, not as an absolute folklore number.

Why does PostgreSQL replication lag spike during vacuums and index builds?

They are WAL-generation bursts (stage 1) and, for redo, page-intensive to apply (stage 4). Schedule them off-peak and watch wal_fpi — post-checkpoint full-page images amplify exactly these operations.

Does logical replication lag behave the same?

No — logical adds walsender decode CPU on the primary and per-subscription apply workers on the subscriber, with different bottlenecks (large-transaction decode, apply-worker serialization). PostgreSQL 16+ parallel apply for large transactions changes that calculus; treat it as a separate diagnosis tree.

Can I make WAL replay parallel?

Not in core PostgreSQL through 17. Mitigate with prefetching (15+), faster standby storage, and by taming primary burst patterns. If a single standby fundamentally can't keep up with sustained write volume, the architecture conversation is sharding or workload splitting, not a parameter.

Slot or no slot for standbys?

Use slots with max_slot_wal_keep_size capped, plus WAL archiving as the recovery backstop. Slots without a cap are a primary-outage risk; archiving without slots makes standby catch-up depend on archive performance. The pair is the production answer.

PostgreSQL Replication Lag Management as a Managed Discipline

MinervaDB designs, audits, and operates PostgreSQL replication topologies — streaming and logical, Patroni-managed HA, cross-region DR with rehearsed failover — under 24×7 SLAs. If PostgreSQL replication lag alerts, slot incidents, or unproven failover paths are on your risk register, engage us: minervadb.com/contact-us · contact@minervadb.com.

Validate every replication parameter change in a staging topology first, and treat failover rehearsal and verified backups as prerequisites to — not outcomes of — any lag-remediation program.