MinervaDB × Google Cloud · Google Cloud Data Platform Engineering · BigQuery, AlloyDB, Cloud SQL, Spanner, Bigtable, Dataflow

Google Cloud Data Platform Engineering: BigQuery, AlloyDB and Spanner Engineered on Evidence

On Google Cloud, the bill and the latency are both decided by engineering details: how many bytes a BigQuery scan stage reads, how many slot-milliseconds a query burns, where AlloyDB keeps its pages and which Spanner split takes every insert. MinervaDB Google Cloud Data Platform Engineering measures those details, designs around them, migrates estates behind gated cutovers and runs the platform 24×7 with a 15-minute Severity-1 response.

15 minSeverity-1 response, 24×7×365
900+enterprises served by MinervaDB
46cities with delivery presence
15+years of database engineering depth
200+years of combined leadership experience

Reference architecture

How Google Cloud data platform engineering lays out the estate

Google Cloud offers at least two credible services for every layer of a data platform, and most estates we inherit use several of them without a rule for which one owns what. Google Cloud Data Platform Engineering starts by assigning each workload to a service from its access pattern, consistency need and measured volume, then puts the whole estate inside one governed perimeter.

Google Cloud Data Platform Engineering reference architecture: sources, Datastream, Pub/Sub and Database Migration Service ingestion, Dataflow and Dataform processing, BigQuery, BigLake, AlloyDB, Spanner and Bigtable storage, Looker and Vertex AI serving, governed by Dataplex, policy tags, VPC Service Controls and Cloud KMS

Figure 1. The Google Cloud data platform engineering reference model: five layers inside a VPC Service Controls perimeter, with catalog, access policy, keys and observability shared across all of them.

Workload Service we default to Why When we choose something else
Enterprise analytics, BI, ELT BigQuery Serverless columnar engine, separation of storage and slots, SQL-first tooling Sub-second, high-concurrency serving on narrow queries: ClickHouse or an AlloyDB columnar read path
Open lakehouse shared with Spark or other engines BigLake Iceberg tables on Cloud Storage One copy of data in an open format, governed from BigQuery Single-engine estates where native BigQuery storage is simpler
PostgreSQL OLTP, write-heavy or mixed analytics AlloyDB for PostgreSQL Disaggregated storage, read pools, columnar engine Small or cost-sensitive databases: Cloud SQL Enterprise edition
MySQL, SQL Server, lift-and-shift PostgreSQL Cloud SQL (Enterprise or Enterprise Plus edition) Familiar engine, managed HA, low operational change Instance-level features Cloud SQL does not expose: self-managed on Compute Engine
Global OLTP with strong consistency Spanner Horizontal scale with external consistency and multi-region configurations Single-region workloads that fit one PostgreSQL primary
Time series, IoT, very high write rates Bigtable Wide-column, single-digit-millisecond key access at scale Workloads that need SQL joins and secondary indexes first

The right-hand column is the one that matters. Google Cloud Data Platform Engineering at MinervaDB is vendor-neutral: when a workload is cheaper or safer on another engine or another cloud, the assessment says so. Our BigQuery consulting and AlloyDB consulting practices carry the engine-level depth behind each row.

Slots and editions

On-demand or BigQuery editions: a capacity decision, measured

On-demand pricing bills bytes scanned; editions bill slot capacity per second. Neither is cheaper in general. The answer depends on the shape of your slot demand across the day, which the timeline views record minute by minute. Google Cloud Data Platform Engineering models both options on that history before recommending a reservation.

Google Cloud Data Platform Engineering BigQuery editions capacity: baseline slots billed continuously, autoscaled slots in multiples of 50 up to the maximum reservation size, and demand above the maximum queuing

Figure 3. The Google Cloud data platform engineering capacity model: baseline, autoscaling and queueing against an illustrative day of demand. Scaling increments and billing minimums follow the slot autoscaling documentation.

GoogleSQL · INFORMATION_SCHEMA.JOBS_TIMELINE: slots used per minute and reservation

-- BigQuery: average slots used per minute and per reservation over the last 24 h
SELECT
  TIMESTAMP_TRUNC(period_start, MINUTE)         AS minute,
  IFNULL(reservation_id, 'on-demand')           AS reservation,
  ROUND(SUM(period_slot_ms) / (1000 * 60), 0)   AS avg_slots_used
FROM `region-us`.INFORMATION_SCHEMA.JOBS_TIMELINE_BY_PROJECT
WHERE job_creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 DAY)
  AND period_start      >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 DAY)
  AND (statement_type IS NULL OR statement_type != 'SCRIPT')
GROUP BY minute, reservation
ORDER BY minute, reservation;

Baseline

In Google Cloud data platform engineering, the baseline is set from the steady floor of the timeline, not from the peak. Commitments price only this part.

Maximum

Caps autoscaling spend. Demand above it queues, so we set it from the latency the business accepts.

Reservations

Google Cloud Data Platform Engineering separates ELT, BI and data science so a backfill cannot starve dashboards; idle slots still flow between them.

Edition

Chosen by required features such as CMEK, VPC Service Controls or managed disaster recovery, then by price.

We rerun the same timeline query a month after the change, so the Google Cloud data platform engineering recommendation is judged by measured slot hours and queue time rather than by the forecast that justified it.

Operational databases

AlloyDB and Cloud SQL, chosen by where the pages live

Cloud SQL runs the database engine on a VM with regional persistent disk; AlloyDB separates PostgreSQL compute from a regional storage service that replays the write-ahead log itself. That difference changes failover mechanics, read scaling and write throughput, so Google Cloud data platform engineering makes the choice from measured write rate, read fan-out and analytical load.

Google Cloud Data Platform Engineering Cloud SQL and AlloyDB architectures: Cloud SQL primary and standby on regional persistent disk with asynchronous read replicas, AlloyDB HA primary and read pool on regional intelligent storage with a log processing service and columnar engine

Figure 4. Cloud SQL high availability on replicated block storage compared with the AlloyDB architecture of disaggregated compute and storage.

Parameter (PostgreSQL) Current Proposed Unit Apply
work_mem 4 32 MB no restart
max_connections 1000 400 + pooler connections restart
autovacuum_vacuum_scale_factor 0.2 0.02 fraction no restart
random_page_cost 4.0 1.1 cost units no restart

Values are illustrative. Each real change is recorded this way from measured workload evidence, applied first in a non-production instance, and scheduled for a maintenance window when the flag restarts the instance.

Google Cloud Data Platform Engineering also covers the MySQL and SQL Server editions of Cloud SQL, with engine depth from our PostgreSQL support and MySQL support teams.

SQL · pg_stat_statements on Cloud SQL or AlloyDB

-- Cloud SQL / AlloyDB for PostgreSQL: top statements by total execution time
-- (pg_stat_statements column names for PostgreSQL 13 and later)
SELECT queryid,
       calls,
       ROUND(total_exec_time::numeric / 1000, 1)        AS total_s,
       ROUND(mean_exec_time::numeric, 2)                AS mean_ms,
       ROUND(100.0 * shared_blks_hit
             / NULLIF(shared_blks_hit + shared_blks_read, 0), 1) AS cache_hit_pct,
       rows,
       LEFT(query, 100)                                 AS query_head
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 20;

In Google Cloud data platform engineering, the same statement ranking runs on both services, which makes it the fairest test when deciding between them: we replay the top statements against a restored copy on each and compare latency, throughput and cost per transaction.

Spanner and Bigtable

Spanner and Bigtable key design that avoids hot splits

Spanner and Bigtable both divide the key space into ranges served by different servers and split ranges under load. A primary key that only grows, such as a sequence or a leading timestamp, sends every insert to the last range, and adding nodes does not help. Google Cloud Data Platform Engineering designs keys from the write pattern and proves the spread with the introspection tables.

Google Cloud Data Platform Engineering Spanner and Bigtable key design: a monotonic key creating a hot split compared with a bit-reversed sequence spreading writes, and Bigtable row keys that lead with the entity

Figure 5. One hot split against an even spread. Score thresholds follow the Spanner hot split statistics; the values shown are illustrative.

GoogleSQL · SPANNER_SYS: warm and hot splits in the last hour

-- Spanner: warm and hot splits in the last hour (CPU_USAGE_SCORE >= 50 is warm, 100 is hot)
SELECT interval_end,
       split_start,
       split_limit,
       cpu_usage_score,
       affected_tables,
       unsplittable_reasons
FROM SPANNER_SYS.SPLIT_STATS_TOP_MINUTE
WHERE interval_end >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 HOUR)
ORDER BY cpu_usage_score DESC, interval_end DESC
LIMIT 50;

For Bigtable, the equivalent Google Cloud data platform engineering evidence is the Key Visualizer heatmap and per-table CPU. A row key such as sensor_id#reversed_timestamp keeps each device’s history contiguous for scans while spreading writes across tablets.

GoogleSQL DDL · bit-reversed sequence and an interleaved child table

-- Spanner GoogleSQL DDL: spread inserts with a bit-reversed sequence, keep children in the parent's split
CREATE SEQUENCE order_id_seq OPTIONS (sequence_kind = 'bit_reversed_positive');

CREATE TABLE orders (
  order_id     INT64      NOT NULL DEFAULT (GET_NEXT_SEQUENCE_VALUE(SEQUENCE order_id_seq)),
  customer_id  STRING(36) NOT NULL,
  created_at   TIMESTAMP  NOT NULL OPTIONS (allow_commit_timestamp = true),
  status       STRING(16) NOT NULL,
) PRIMARY KEY (order_id);

CREATE TABLE order_items (
  order_id  INT64      NOT NULL,
  line_no   INT64      NOT NULL,
  sku       STRING(64) NOT NULL,
  qty       INT64      NOT NULL,
) PRIMARY KEY (order_id, line_no),
  INTERLEAVE IN PARENT orders ON DELETE CASCADE;

Migration

Moving databases and warehouses onto Google Cloud

Every migration in a Google Cloud data platform engineering programme follows the same discipline: replicate continuously, reconcile, and cut over behind a written gate with a rollback path. The tooling differs by source.

Source Target Path What we reconcile
PostgreSQL (any host) AlloyDB or Cloud SQL Database Migration Service, continuous job Row counts, checksums, sequences, extensions
MySQL Cloud SQL for MySQL Database Migration Service, continuous job Row counts, checksums, users and grants
Oracle AlloyDB or Cloud SQL for PostgreSQL Database Migration Service conversion workspace, then CDC Converted PL/SQL, data types, row counts
SQL Server Cloud SQL for SQL Server Database Migration Service from backups Logins, jobs, row counts
Teradata, Redshift, Snowflake, Hive BigQuery BigQuery Migration Service SQL translation, BigQuery Data Transfer Service Report-level results, aggregates per partition
Operational change data BigQuery Datastream CDC Freshness and row-level parity

Step 1

Assess

Inventory, compatibility, extensions, sizing from measured load.

Step 2

Replicate

Google Cloud Data Platform Engineering runs a full load, then change capture with lag alerting.

Step 3

Reconcile

Row counts and checksums per table; replay of top statements.

Step 4

Cut over

Writes stopped, lag at zero, typed confirmation, named approver.

Rollback is planned before cutover, not after: the source stays intact, and for PostgreSQL targets we can set up reverse logical replication so the old primary can take traffic back. The Database Migration Service documentation covers supported sources and versions, which we confirm per engagement.

Bash · gcloud: verify, confirm, promote, validate

#!/usr/bin/env bash
# Cutover gate for a Database Migration Service continuous job (for example PostgreSQL to AlloyDB)
set -euo pipefail
: "${PROJECT_ID:?set PROJECT_ID}" "${REGION:?set REGION}" "${MIGRATION_JOB:?set MIGRATION_JOB}"

# 1. Verify before: the job is RUNNING and in the CDC phase
gcloud database-migration migration-jobs describe "${MIGRATION_JOB}" \
  --region="${REGION}" --project="${PROJECT_ID}" \
  --format="value(state,phase)"            # expected: RUNNING  CDC

# 2. Confirmation gate: application writes stopped, replication lag at zero in Cloud Monitoring
read -r -p "Writes stopped and lag at 0 for ${MIGRATION_JOB}? Type the job name to promote: " answer
if [[ "${answer}" != "${MIGRATION_JOB}" ]]; then
  echo "Aborted. No change made."; exit 1
fi

# 3. Promote the destination. The source stops replicating to it from this point.
gcloud database-migration migration-jobs promote "${MIGRATION_JOB}" \
  --region="${REGION}" --project="${PROJECT_ID}"

# 4. Validate after: the job reports completion; then run row-count and checksum reconciliation
gcloud database-migration migration-jobs describe "${MIGRATION_JOB}" \
  --region="${REGION}" --project="${PROJECT_ID}" \
  --format="value(state,phase)"            # expected: COMPLETED once promotion finishes

Governance and security

Governance that is enforced, not documented

Auditors need evidence; engineers need to ship. In Google Cloud data platform engineering, each obligation maps to a control Google Cloud enforces and to a query or report that proves it.

Obligation Google Cloud control How we prove it
Least privilege IAM groups, predefined roles, no primitive roles on data projects IAM policy export, Policy Analyzer
Exfiltration control VPC Service Controls perimeter around data projects Perimeter configuration and dry-run violation logs
Column and row security BigQuery policy tags, row access policies, dynamic data masking Taxonomy inventory, policy listing per table
Encryption CMEK with Cloud KMS for BigQuery, AlloyDB, Cloud SQL and Spanner Key usage per resource, key rotation records
Catalog, lineage, quality Dataplex catalog, lineage and data quality scans Lineage for audited reports, quality scan results
Sensitive data discovery Sensitive Data Protection profiling Profile findings mapped to policy tags
Regulation and residency Organization policy for resource locations; SOC 2, ISO 27001, HIPAA, GDPR, India’s DPDP Act Control-to-evidence map per framework

Where Google Cloud data platform engineering governance spans more than Google Cloud, the design runs with our data governance consulting team; model pipelines on Vertex AI follow the controls of our MLOps consulting practice.

Test every schema change, reservation change, flag change, policy and cutover in a non-production project first, keep backups and point-in-time recovery verified by restore drills, and maintain a robust disaster-recovery posture with a tested secondary region.

Engagement model

Assessment, build and 24×7 operations

Every Google Cloud data platform engineering engagement begins with two weeks of measurement across job history, slot timelines, database statistics and billing export. The output is a ranked list of findings with the evidence query behind each one.

Weeks 1–2

Measure

Query shapes, slot demand, database hotspots, spend attribution and risk register.

Design

Decide

Service placement, capacity model, key and schema designs, migration waves and rollback criteria.

Build

Deliver

The Google Cloud data platform engineering team builds, migrates and tunes, with a verification query after every change.

Run

Operate

24×7 operations under SLA, monthly cost and performance reviews, quarterly DR drills.

Severity 115 minProduction down, data at risk or failover in progress. 24×7×365.
Severity 212 hSevere degradation, throttling or queueing with no workaround.
Severity 324 hDegradation with a workaround in place.
Severity 448 hQuestions, planning and advisory requests.

Related services: remote DBA services, data analytics and data warehousing support and 24/7 emergency DBA coverage.

FAQ

Google Cloud Data Platform Engineering questions we hear most

Short answers to the questions data leaders ask before a Google Cloud data platform engineering engagement.

Should we use BigQuery on-demand pricing or editions?

It depends on the shape of your slot demand. In Google Cloud data platform engineering work, steady, heavy usage usually favours editions with a baseline and autoscaling; spiky or light usage often stays cheaper on-demand. We model both from INFORMATION_SCHEMA.JOBS_TIMELINE before recommending either.

Why is our BigQuery bill growing faster than our data?

Usually a small number of recurring query shapes: dashboards scanning unpartitioned tables, SELECT * on wide tables, or scheduled queries that bypass the cache. We rank shapes by bytes billed and slot time and fix the top ones first.

AlloyDB or Cloud SQL for PostgreSQL?

Cloud SQL suits standard OLTP and cost-sensitive databases. AlloyDB suits write-heavy PostgreSQL, large read fan-out and mixed analytical queries. We replay your top statements on both and compare latency and cost per transaction.

When is Spanner the right choice?

When you need horizontal write scale or multi-region strong consistency that one PostgreSQL primary cannot give. For single-region workloads that fit one primary, AlloyDB or Cloud SQL is usually simpler and cheaper.

Why does adding Spanner nodes not fix our write latency?

Most often a monotonic primary key sends every insert to one split. We confirm it in SPANNER_SYS.SPLIT_STATS_TOP_MINUTE and redesign the key with a bit-reversed sequence, UUIDv4 or a hashed prefix.

How do you migrate without a long outage?

Google Cloud Data Platform Engineering uses continuous replication through Database Migration Service or Datastream, reconciliation by row counts and checksums, and a cutover gate that requires stopped writes, zero lag, a typed confirmation and a prepared rollback path.

Do you only design, or also operate the platform?

Both. Most Google Cloud data platform engineering engagements continue into 24×7 managed operations with a 15-minute Severity-1 response, monthly reviews and quarterly disaster-recovery drills.

Will you tell us when Google Cloud is the wrong answer?

Yes. We are vendor-neutral and do not resell Google Cloud. When a workload is cheaper or safer on another engine or cloud, we say so in writing before any project starts.

Make every slot and every byte on Google Cloud count

Start with a two-week assessment of query shapes, slot demand, database hotspots and spend, ranked by impact. Google Cloud Data Platform Engineering for active incidents is available through the same page, 24×7.