Cloud Native Database Support

Two consequences follow for cloud native database support. Manual intervention is temporary, because anything changed by hand is reverted unless the resource that describes it changes too. And an operator upgrade is a production change, because the reconciliation logic is what decides whether your primary is fenced correctly. We pin operator versions, rehearse operator upgrades in a staging cluster with the production storage class, and keep them on a separate calendar from engine upgrades. The Kubernetes operator pattern describes the mechanism.

Our operator rubric has five questions. Does it fence before it promotes? Does it restore to a timestamp rather than a segment boundary? Can it roll a minor version without a rebuild? Does it expose replication lag and archive age as status conditions? Is the project openly governed with a published support window? The manifest on the right is the baseline Cloud Native Database Support starts from on CloudNativePG, with quorum synchronous replication, separate WAL storage, required zone anti-affinity and Guaranteed memory.

# CloudNativePG Cluster (operator 1.30.x, Kubernetes 1.37). Example values: size from your own baseline.
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: orders-pg
  namespace: db-orders
spec:
  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:17.6     # pin a digest in production
  primaryUpdateStrategy: unsupervised
  postgresql:
    synchronous:
      method: any                 # quorum: ANY 1 of the standbys must confirm
      number: 1
      dataDurability: required    # refuse writes rather than silently go async
    parameters:
      shared_buffers: "8GB"
      max_connections: "300"      # the pooler fans in; backends cost cgroup memory
  storage:
    storageClass: db-gp3-wffc
    size: 400Gi
  walStorage:
    storageClass: db-gp3-wffc
    size: 100Gi
  affinity:
    enablePodAntiAffinity: true
    topologyKey: topology.kubernetes.io/zone
    podAntiAffinityType: required
  resources:
    requests: {cpu: "4", memory: "32Gi"}
    limits:   {memory: "32Gi"}    # memory request = limit; no CPU limit (see throttling)
  superuserSecret:
    name: orders-pg-superuser     # created from your secret manager, never committed

Cloud Native Database Support game days record three numbers that are routinely confused. Detection time is how long the platform takes to believe the primary is gone, and it is set by lease and probe settings. Promotion time runs from a completed fence to the first write accepted on the new primary. Application recovery time runs from the first client error to the first successful transaction, and includes endpoint propagation, pool eviction and retry backoff. The last number is the one your users feel, and the one a vendor datasheet never quotes.

Synchronous quorum replication decides what a failover can lose. With method: any and number: 1, a commit is acknowledged only after one standby confirms it, so the best candidate holds every acknowledged transaction. With dataDurability: required, the primary stops accepting writes rather than silently degrading to asynchronous when no standby is available. That is a deliberate availability-for-durability trade we agree with each client in writing. When an outage is already in progress, our 24/7 emergency DBA coverage takes the call.

Planned switchovers run through a script with a verification step before, a typed confirmation gate, and validation after. Rollback is the same script targeting the previous primary once it is streaming again.

#!/usr/bin/env bash
# Planned switchover with a confirmation gate. Requires the cnpg kubectl plugin matching the operator version.
set -euo pipefail
NS=db-orders; CL=orders-pg; TARGET=${1:?usage: switchover.sh <target-instance>}

# 1. Verify before: topology, lag and archive health, queried on the current primary
PRIMARY=$(kubectl get cluster "$CL" -n "$NS" -o jsonpath='{.status.currentPrimary}')
kubectl cnpg status "$CL" -n "$NS"
kubectl exec -n "$NS" "$PRIMARY" -c postgres -- psql -At -c \
  "SELECT application_name, sync_state, pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS lag_bytes FROM pg_stat_replication;"

# 2. Confirmation gate: nothing changes without an explicit, typed answer
read -r -p "Promote ${TARGET} in ${NS}/${CL}? Type the cluster name to continue: " ANSWER
[[ "$ANSWER" == "$CL" ]] || { echo "aborted, no change made"; exit 1; }

# 3. Act
kubectl cnpg promote "$CL" "$TARGET" -n "$NS"

# 4. Validate after: new primary, rw Service endpoint, replicas streaming again
kubectl get cluster "$CL" -n "$NS" -o jsonpath='{.status.currentPrimary}{"\n"}'
kubectl get endpointslices -n "$NS" -l kubernetes.io/service-name="${CL}-rw" -o wide
kubectl cnpg status "$CL" -n "$NS"
# Rollback: run this script again with the previous primary as TARGET once it is streaming.

Page cache is charged to the container's cgroup, so a database that leans on the OS cache can be evicted for memory it would happily give back. Cloud Native Database Support sizes the engine cache explicitly and alerts on throttling counters rather than on average CPU. The mechanics are in the Kubernetes resource management guide; since Kubernetes 1.35, in-place pod resize lets us raise CPU without a restart, where the operator supports it.

# Share of CFS periods in which a database container was throttled (cAdvisor metrics)
sum by (namespace, pod, container) (
  rate(container_cpu_cfs_throttled_periods_total{namespace=~"db-.*", container="postgres"}[5m])
)
/
sum by (namespace, pod, container) (
  rate(container_cpu_cfs_periods_total{namespace=~"db-.*", container="postgres"}[5m])
)
> 0.05   # alert: more than 5% of periods throttled, sustained for 15 minutes

# Working set against the memory limit: the OOM-kill early warning
max by (pod) (container_memory_working_set_bytes{namespace=~"db-.*", container="postgres"})
/
max by (pod) (kube_pod_container_resource_limits{namespace=~"db-.*", container="postgres", resource="memory"})

Auditability is a cloud native database support deliverable. Every privileged action, including an interactive shell into a database pod, produces a durable log entry outside the cluster, retained for the period your regulator expects.

# Default-deny for the database namespace, then one explicit allow: pooler -> database on 5432
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: db-orders
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-pooler-to-postgres
  namespace: db-orders
spec:
  podSelector:
    matchLabels: {cnpg.io/cluster: orders-pg}
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels: {cnpg.io/poolerName: orders-pg-pooler-rw}
      ports:
        - {protocol: TCP, port: 5432}
# Replication between instances, operator status checks, DNS and archive egress need their own allows:
# write them before applying default-deny, and test in staging first.