MinervaDB × Microsoft Azure · Azure Data Platform Engineering · Azure SQL, Managed Instance, PostgreSQL, Cosmos DB, Microsoft Fabric

Azure Data Platform Engineering: Azure SQL, Cosmos DB and Microsoft Fabric, Engineered From the Storage Layer Up

On Azure, the service tier is a storage architecture, the Cosmos DB partition key is a throughput budget, and Fabric capacity is a smoothing window that can throttle tomorrow’s dashboards for tonight’s load. MinervaDB Azure Data Platform Engineering makes those decisions with measured evidence, migrates SQL Server and Oracle estates with a gated cutover, and runs the platform 24×7 with a 15-minute Severity-1 response.

15 minSeverity-1 response, 24×7×365
900+enterprises served by MinervaDB
46cities with delivery presence
15+years of database engineering depth
200+years of combined leadership experience

Foundation

The data landing zone Azure Data Platform Engineering builds first

Most Azure data incidents we inherit trace back to the foundation: a database with public network access, a pipeline authenticating with a stored key, a lake nobody catalogued. Azure Data Platform Engineering starts with a landing zone where identity, keys, policy and catalog are platform services, and every database, lake and pipeline sits in a spoke reached only through private endpoints.

Azure Data Platform Engineering landing zone: management groups and Azure Policy, hub virtual network with firewall and private DNS, operational, analytics and ingestion spokes reached through private endpoints, with Entra ID, Key Vault, Purview and Defender for Cloud

Figure 1. The data landing zone: platform services on the left, a hub for connectivity and DNS, and three data spokes whose services are reachable only through private endpoints.

Azure Policy does the enforcement, so governance does not depend on every engineer remembering it. We assign deny policies for public network access, require customer-managed keys where regulation asks for them, restrict regions for residency, and let pipelines authenticate with managed identities rather than secrets. The Cloud Adoption Framework for cloud-scale analytics describes the pattern; Azure data platform engineering adapts it to the estate you actually have.

The query alongside is the first check we run in a new tenant. It lists every data service that still accepts traffic from public networks, across all subscriptions, in seconds. The result is usually the first finding in an Azure data platform engineering assessment.

KQL · Azure Resource Graph: data services with public network access enabled

// Azure Resource Graph: data services still reachable from public networks (run across all subscriptions)
resources
| where type in~ ('microsoft.sql/servers',
                  'microsoft.dbforpostgresql/flexibleservers',
                  'microsoft.dbformysql/flexibleservers',
                  'microsoft.documentdb/databaseaccounts',
                  'microsoft.storage/storageaccounts')
| extend pna = tostring(coalesce(properties.publicNetworkAccess,
                                 properties.network.publicNetworkAccess))
| where pna =~ 'Enabled'
| project subscriptionId, resourceGroup, name, type, location, pna
| order by type asc, name asc

Azure SQL

Azure SQL service tiers are storage architectures

General Purpose, Business Critical and Hyperscale are not bigger and smaller versions of the same thing. They place data and log on different storage, fail over differently and scale reads differently. Azure Data Platform Engineering chooses the tier from measured I/O latency, log generation rate, failover tolerance and growth, then tunes the engine with Query Store evidence.

Azure Data Platform Engineering Azure SQL Database tier architectures: General Purpose with remote storage, Business Critical with local SSD and an availability group, and Hyperscale with page servers and a log service

Figure 2. General Purpose, Business Critical and Hyperscale compared by storage path, replicas and failover mechanics, as assessed in every Azure data platform engineering review.

Signal Where we read it What it decides
I/O latency on data and log sys.dm_io_virtual_file_stats General Purpose vs Business Critical
Log generation rate sys.dm_db_resource_stats (avg_log_write_percent) Log-rate ceiling per tier and size
Wait profile sys.dm_db_wait_stats, Query Store waits CPU, I/O or lock problem first
Plan regressions Query Store runtime stats Forced plans vs code or index fixes
Size and growth allocated vs used space trend Hyperscale when growth is steep

Serverless compute with auto-pause suits intermittent databases; Business Critical suits latency-sensitive OLTP; Hyperscale suits large or fast-growing databases and read scale-out through named replicas. For SQL Server compatibility at instance scope, Azure SQL Managed Instance keeps Agent jobs, cross-database queries and CLR. Engine-level depth comes from our SQL Server support practice.

T-SQL · Query Store regression finder (Azure SQL Database and Managed Instance)

-- Query Store: queries whose average duration in the last 24 h regressed against the prior 7 days
WITH w AS (
    SELECT p.query_id,
           CASE WHEN i.start_time >= DATEADD(HOUR, -24, SYSUTCDATETIME()) THEN 'recent' ELSE 'baseline' END AS win,
           rs.avg_duration * rs.count_executions AS dur_us,          -- avg_duration is in microseconds
           rs.count_executions
    FROM sys.query_store_runtime_stats          AS rs
    JOIN sys.query_store_runtime_stats_interval AS i ON i.runtime_stats_interval_id = rs.runtime_stats_interval_id
    JOIN sys.query_store_plan                   AS p ON p.plan_id = rs.plan_id
    WHERE i.start_time >= DATEADD(DAY, -8, SYSUTCDATETIME())
),
agg AS (
    SELECT query_id,
           SUM(CASE WHEN win = 'recent'   THEN dur_us END) / NULLIF(SUM(CASE WHEN win = 'recent'   THEN count_executions END), 0) AS recent_us,
           SUM(CASE WHEN win = 'baseline' THEN dur_us END) / NULLIF(SUM(CASE WHEN win = 'baseline' THEN count_executions END), 0) AS base_us,
           SUM(CASE WHEN win = 'recent'   THEN count_executions END) AS recent_execs
    FROM w
    GROUP BY query_id
)
SELECT TOP (20)
       a.query_id,
       CAST(a.base_us   / 1000.0 AS DECIMAL(12,2)) AS baseline_ms,
       CAST(a.recent_us / 1000.0 AS DECIMAL(12,2)) AS recent_ms,
       a.recent_execs,
       LEFT(qt.query_sql_text, 120)                AS query_head
FROM agg AS a
JOIN sys.query_store_query      AS q  ON q.query_id = a.query_id
JOIN sys.query_store_query_text AS qt ON qt.query_text_id = q.query_text_id
WHERE a.recent_us > 2 * a.base_us
  AND a.recent_execs > 100
ORDER BY (a.recent_us - a.base_us) * a.recent_execs DESC;

Open-source engines

Azure Database for PostgreSQL and MySQL flexible servers

Single Server is retired for both engines, so every open-source estate on Azure now runs on Flexible Server. Azure Data Platform Engineering treats it as the managed engine it is: we still own the query plans, vacuum, connection pooling and version calendar, while Azure owns the host, patching windows and zone-redundant standby.

Zone-redundant HA

In Azure data platform engineering we run a synchronous standby in another availability zone with automatic failover. We measure real failover time from the client side and tune retry logic, because the application sees more than the service promises.

Built-in PgBouncer

For Azure data platform engineering, Flexible Server ships PgBouncer on the server, so connection storms from App Service or AKS do not exhaust backends. We size pool mode and limits from measured concurrency.

Server parameters

shared_buffers, work_mem, autovacuum and max_connections set from the workload, with the change recorded as current value, proposed value, unit and restart requirement.

Read replicas and virtual endpoints

Read scale-out without application rewrites, with replica lag alerting and a promotion runbook tested before it is needed.

Extensions allowlist

pg_stat_statements, pgvector, pg_cron and PostGIS enabled through azure.extensions, checked before migration rather than after.

Version calendar

Major upgrades planned with in-place upgrade or a logical replica, tested on a restored copy, so retirement dates never become incidents.

Deeper engine work runs through our PostgreSQL support and MySQL support teams, with the same Azure data platform engineering on-call covering both. Microsoft publishes platform changes in the Flexible Server release notes, which we track monthly.

Analytics

Microsoft Fabric capacity, OneLake and the Synapse path

Fabric runs every workload against a shared capacity measured in capacity units, and it smooths consumption rather than rejecting spikes immediately. That is convenient until the carried-forward overage catches up and interactive reports start to queue. Azure Data Platform Engineering sizes and schedules Fabric capacity from the Capacity Metrics app, not from the SKU catalogue.

Azure Data Platform Engineering Microsoft Fabric capacity smoothing and throttling: raw and smoothed capacity unit consumption, carried-forward overage, and the stages that delay then reject interactive and background work

Figure 4. Fabric smoothing and the throttling stages from the Fabric throttling policy; consumption values are illustrative.

For Azure data platform engineering, the key fact is that background operations such as pipeline runs, Spark jobs and dataflow refreshes are smoothed over 24 hours, while interactive queries are smoothed over a much shorter window. A heavy nightly load can therefore still be charged against capacity the next morning. Azure Data Platform Engineering separates capacities by workload where the metrics justify it, moves heavy jobs, and pre-aggregates so Direct Lake semantic models read less.

Under Azure data platform engineering, estates still on Azure Synapse dedicated SQL pools get a planned path: tune distribution and partitioning where the pool stays, or mirror and migrate to the Fabric Warehouse where it does not. Either way, the query ranking alongside is how we find the statements that consume the capacity.

T-SQL · Fabric Warehouse: statement shapes ranked by total elapsed time

-- Fabric Data Warehouse: statement shapes ranked by total elapsed time over 7 days (queryinsights views)
SELECT TOP (20)
       query_hash,
       COUNT(*)                                        AS executions,
       CAST(SUM(total_elapsed_time_ms) / 1000.0 AS DECIMAL(12,1)) AS total_s,
       CAST(SUM(allocated_cpu_time_ms) / 1000.0 AS DECIMAL(12,1)) AS cpu_s,
       SUM(data_scanned_remote_storage_mb)             AS remote_scan_mb,
       LEFT(MAX(command), 120)                         AS sample_statement
FROM queryinsights.exec_requests_history
WHERE start_time >= DATEADD(DAY, -7, GETUTCDATE())
  AND status = 'Succeeded'
GROUP BY query_hash
ORDER BY total_s DESC;

Migration

SQL Server to Azure with a gated, reversible cutover

An ageing SQL Server instance nobody dares touch is the most common starting point we see. Azure Data Platform Engineering moves it with replication rather than a weekend outage: the Managed Instance link keeps a readable copy synchronized for weeks, the application team validates against it, and cutover becomes a planned failover behind a written gate.

Azure Data Platform Engineering SQL Server to Azure SQL Managed Instance migration with the Managed Instance link: distributed availability group replication, validation, a gated planned failover and reverse replication for rollback

Figure 5. The five-step migration path through the Managed Instance link, with the cutover gate and the rollback direction.

T-SQL · cutover gate: verify synchronization and empty queues on the source

-- 1. Verify on the SQL Server source: the link's database replicas are synchronized and queues are empty
SELECT ag.name                         AS availability_group,
       ag.is_distributed,
       DB_NAME(drs.database_id)        AS database_name,
       drs.synchronization_state_desc,
       drs.log_send_queue_size         AS log_send_queue_kb,
       drs.redo_queue_size             AS redo_queue_kb,
       drs.last_commit_time
FROM sys.dm_hadr_database_replica_states AS drs
JOIN sys.availability_groups             AS ag ON ag.group_id = drs.group_id
WHERE drs.is_local = 0;

Oracle, MySQL and PostgreSQL sources follow the same principle with Azure Database Migration Service in online mode: full load, change capture, row-count and checksum reconciliation, then a gate. We favour incremental cutovers over big-bang weekends in every Azure data platform engineering plan.

PowerShell · confirmation gate, planned failover and validation

# 2. Confirmation gate, then planned failover of the Managed Instance link (Az.Sql module)
$rg   = $env:MI_RESOURCE_GROUP;  $mi = $env:MI_NAME;  $link = $env:MI_LINK_NAME

$answer = Read-Host "Writes stopped and both queues at 0 for $link? Type the link name to fail over"
if ($answer -ne $link) { Write-Host "Aborted. No change made."; exit 1 }

Start-AzSqlInstanceLinkFailover -ResourceGroupName $rg -InstanceName $mi -Name $link -FailoverType Planned

# 3. Validate: the managed instance is now primary for the link; then repoint connection strings
Get-AzSqlInstanceLink -ResourceGroupName $rg -InstanceName $mi -Name $link | Format-List *

Cost engineering

Where Azure data platform engineering finds Azure data spend

An Azure bill that surprises the CFO almost always traces back to engineering choices. We attribute spend from Cost Management exports to workloads and owners, then fix the causes and verify the savings with the same query a month later.

Service Where spend hides What we change Evidence
Azure SQL Database vCores sized for a rare peak; Business Critical where latency does not need it Right-size from measured load; serverless with auto-pause for intermittent databases; reservations for the steady baseline sys.dm_db_resource_stats, Cost Management
Managed Instance Idle development instances; licence not using Azure Hybrid Benefit Stop/start schedules where supported; hybrid benefit where licences allow Instance utilisation, licence inventory
Cosmos DB Provisioned RU/s far above consumed; cross-partition queries Autoscale or serverless; key redesign; indexing policy trimmed Normalized RU consumption, diagnostic logs
Microsoft Fabric Capacity sized for a twice-yearly peak; one capacity for every workload Split capacities; reschedule background jobs; pause non-production Capacity Metrics app
Synapse dedicated pools Pools online over weekends; replicated tables that should be hash-distributed Pause schedules; distribution fixes; Fabric migration plan DMV skew and data movement steps
Storage and network Premium disks and geo-replication where not required; egress between regions Tier and replication review; co-locate compute and data Cost by meter and resource

Beyond Azure data platform engineering reviews, programme-level cost work runs under our cloud database optimization and FinOps practice, alongside the AWS and Google Cloud data platform engineering teams for multi-cloud estates.

Governance and security

Governance auditors and engineers both accept

In Azure data platform engineering, security teams want provable control; engineers want to ship without friction. The Azure governance stack, assembled well, gives both, and Azure data platform engineering maps each obligation to an enforced control rather than a policy document.

Obligation Azure control How we prove it
Identity and least privilege Microsoft Entra ID authentication, PIM for admin roles, managed identities, contained users Role assignment inventory, PIM activation logs
Network isolation Private endpoints, public access denied by Azure Policy Resource Graph query, policy compliance state
Encryption TDE with customer-managed keys in Key Vault; TLS 1.2+ enforced Key Vault audit, server TLS settings
Data protection in the engine Row-level security, dynamic data masking, Always Encrypted where required Policy and masking inventory per database
Catalog and lineage Microsoft Purview scans, classification, lineage from pipelines Classified-asset reports, lineage for audited reports
Audit and threat detection Azure SQL auditing to Log Analytics, Defender for SQL KQL queries answering who read what, and alerts triaged
Regulation and residency Region policy, geo-replication scoping; SOC 2, ISO 27001, HIPAA, GDPR, India’s DPDP Act Control-to-evidence map per framework

Test every tier change, parameter change, policy assignment and failover in a non-production subscription first, keep point-in-time restore and geo-backups verified by restore drills, and maintain a robust disaster-recovery posture with a tested secondary region.

Engagement model

From assessment to 24×7 operations

We meet the estate where it is: a greenfield Fabric build, a stalled SQL Server migration, or a platform that works but costs too much. Every Azure data platform engineering engagement starts with two weeks of measurement, because evidence beats a quarter of strategy decks.

Phase 1 · weeks 1–2

Assess

The Azure data platform engineering assessment covers landing-zone gaps, tier fit, Cosmos DB keys, Fabric capacity, spend and risks, ranked and costed.

Phase 2

Architect

Target estate, tier and capacity decisions, network and identity design, migration waves and rollback criteria.

Phase 3

Engineer

Build, migrate and tune with senior engineers, validated at every cutover gate.

Phase 4

Operate

24×7 operations under SLA, monthly cost and performance reviews, quarterly DR drills.

Severity 115 minProduction down, failover in progress or data at risk. 24×7×365.
Severity 212 hSevere degradation or throttling with no workaround.
Severity 324 hDegradation with a workaround in place.
Severity 448 hQuestions, planning and advisory requests.

Related services: remote DBA services, data analytics and data warehousing support and 24/7 emergency DBA coverage.

FAQ

Questions about Azure Data Platform Engineering with MinervaDB

The questions data leaders ask most often before an Azure data platform engineering engagement.

Which Azure SQL tier should we use?

It depends on measured I/O latency, log rate, failover tolerance and growth. General Purpose suits cost-sensitive, latency-tolerant work; Business Critical suits OLTP that needs the lowest latency; Hyperscale suits large or fast-growing databases and read scale-out.

Azure SQL Database or Managed Instance?

Azure SQL Database for new, database-scoped applications; Managed Instance when you need instance-level SQL Server features such as Agent jobs, cross-database queries or CLR, or a near-lift-and-shift migration through the Managed Instance link.

How do you migrate SQL Server without a big-bang weekend?

With the Managed Instance link or online Azure Database Migration Service: replicate for weeks, validate on the target, then cut over behind a gate that requires empty send and redo queues, a typed confirmation and a prepared rollback path.

Why is our Cosmos DB throttling when average RU usage is low?

It is the most common Azure data platform engineering finding on Cosmos DB. Provisioned throughput is divided across physical partitions, so a hot partition key hits its share while others sit idle. We find the key in diagnostic logs and redesign it with higher cardinality, hierarchical or synthetic keys.

Why do Fabric reports slow down in the morning?

Background jobs are smoothed over 24 hours, so a heavy overnight load can push the capacity into carry-forward overage and delay interactive work. We reschedule, split capacities and pre-aggregate based on the Capacity Metrics app.

Should we stay on Synapse dedicated pools or move to Fabric?

It depends on workload, feature use and cost. We tune pools that should stay and plan mirroring and migration to the Fabric Warehouse where it pays off, with measured before-and-after numbers.

Do you only consult, or do you also run the platform?

Both. Most Azure data platform engineering engagements continue into 24×7 managed operations with a 15-minute Severity-1 response, monthly reviews and quarterly disaster-recovery drills.

Will you tell us when Azure is the wrong answer?

Yes. We are vendor-neutral and do not resell Azure. When a workload is cheaper or safer self-managed, on another cloud or on another engine, we say so in writing before any project starts.

Build an Azure data estate that performs and pays off

Start with a two-week assessment of landing zone, tiers, partitioning, capacity and spend, ranked by impact. Azure Data Platform Engineering for active incidents is available through the same page, 24×7.