MinervaDB × Microsoft Azure · Azure Data Platform Engineering · Azure SQL, Managed Instance, PostgreSQL, Cosmos DB, Microsoft Fabric
Azure Data Platform Engineering: Azure SQL, Cosmos DB and Microsoft Fabric, Engineered From the Storage Layer Up
On Azure, the service tier is a storage architecture, the Cosmos DB partition key is a throughput budget, and Fabric capacity is a smoothing window that can throttle tomorrow’s dashboards for tonight’s load. MinervaDB Azure Data Platform Engineering makes those decisions with measured evidence, migrates SQL Server and Oracle estates with a gated cutover, and runs the platform 24×7 with a 15-minute Severity-1 response.
Foundation
The data landing zone Azure Data Platform Engineering builds first
Most Azure data incidents we inherit trace back to the foundation: a database with public network access, a pipeline authenticating with a stored key, a lake nobody catalogued. Azure Data Platform Engineering starts with a landing zone where identity, keys, policy and catalog are platform services, and every database, lake and pipeline sits in a spoke reached only through private endpoints.
Figure 1. The data landing zone: platform services on the left, a hub for connectivity and DNS, and three data spokes whose services are reachable only through private endpoints.
Azure Policy does the enforcement, so governance does not depend on every engineer remembering it. We assign deny policies for public network access, require customer-managed keys where regulation asks for them, restrict regions for residency, and let pipelines authenticate with managed identities rather than secrets. The Cloud Adoption Framework for cloud-scale analytics describes the pattern; Azure data platform engineering adapts it to the estate you actually have.
The query alongside is the first check we run in a new tenant. It lists every data service that still accepts traffic from public networks, across all subscriptions, in seconds. The result is usually the first finding in an Azure data platform engineering assessment.
KQL · Azure Resource Graph: data services with public network access enabled
// Azure Resource Graph: data services still reachable from public networks (run across all subscriptions)
resources
| where type in~ ('microsoft.sql/servers',
'microsoft.dbforpostgresql/flexibleservers',
'microsoft.dbformysql/flexibleservers',
'microsoft.documentdb/databaseaccounts',
'microsoft.storage/storageaccounts')
| extend pna = tostring(coalesce(properties.publicNetworkAccess,
properties.network.publicNetworkAccess))
| where pna =~ 'Enabled'
| project subscriptionId, resourceGroup, name, type, location, pna
| order by type asc, name asc
Azure SQL
Azure SQL service tiers are storage architectures
General Purpose, Business Critical and Hyperscale are not bigger and smaller versions of the same thing. They place data and log on different storage, fail over differently and scale reads differently. Azure Data Platform Engineering chooses the tier from measured I/O latency, log generation rate, failover tolerance and growth, then tunes the engine with Query Store evidence.
Figure 2. General Purpose, Business Critical and Hyperscale compared by storage path, replicas and failover mechanics, as assessed in every Azure data platform engineering review.
| Signal | Where we read it | What it decides |
|---|---|---|
| I/O latency on data and log | sys.dm_io_virtual_file_stats |
General Purpose vs Business Critical |
| Log generation rate | sys.dm_db_resource_stats (avg_log_write_percent) |
Log-rate ceiling per tier and size |
| Wait profile | sys.dm_db_wait_stats, Query Store waits |
CPU, I/O or lock problem first |
| Plan regressions | Query Store runtime stats | Forced plans vs code or index fixes |
| Size and growth | allocated vs used space trend | Hyperscale when growth is steep |
Serverless compute with auto-pause suits intermittent databases; Business Critical suits latency-sensitive OLTP; Hyperscale suits large or fast-growing databases and read scale-out through named replicas. For SQL Server compatibility at instance scope, Azure SQL Managed Instance keeps Agent jobs, cross-database queries and CLR. Engine-level depth comes from our SQL Server support practice.
T-SQL · Query Store regression finder (Azure SQL Database and Managed Instance)
-- Query Store: queries whose average duration in the last 24 h regressed against the prior 7 days
WITH w AS (
SELECT p.query_id,
CASE WHEN i.start_time >= DATEADD(HOUR, -24, SYSUTCDATETIME()) THEN 'recent' ELSE 'baseline' END AS win,
rs.avg_duration * rs.count_executions AS dur_us, -- avg_duration is in microseconds
rs.count_executions
FROM sys.query_store_runtime_stats AS rs
JOIN sys.query_store_runtime_stats_interval AS i ON i.runtime_stats_interval_id = rs.runtime_stats_interval_id
JOIN sys.query_store_plan AS p ON p.plan_id = rs.plan_id
WHERE i.start_time >= DATEADD(DAY, -8, SYSUTCDATETIME())
),
agg AS (
SELECT query_id,
SUM(CASE WHEN win = 'recent' THEN dur_us END) / NULLIF(SUM(CASE WHEN win = 'recent' THEN count_executions END), 0) AS recent_us,
SUM(CASE WHEN win = 'baseline' THEN dur_us END) / NULLIF(SUM(CASE WHEN win = 'baseline' THEN count_executions END), 0) AS base_us,
SUM(CASE WHEN win = 'recent' THEN count_executions END) AS recent_execs
FROM w
GROUP BY query_id
)
SELECT TOP (20)
a.query_id,
CAST(a.base_us / 1000.0 AS DECIMAL(12,2)) AS baseline_ms,
CAST(a.recent_us / 1000.0 AS DECIMAL(12,2)) AS recent_ms,
a.recent_execs,
LEFT(qt.query_sql_text, 120) AS query_head
FROM agg AS a
JOIN sys.query_store_query AS q ON q.query_id = a.query_id
JOIN sys.query_store_query_text AS qt ON qt.query_text_id = q.query_text_id
WHERE a.recent_us > 2 * a.base_us
AND a.recent_execs > 100
ORDER BY (a.recent_us - a.base_us) * a.recent_execs DESC;
Open-source engines
Azure Database for PostgreSQL and MySQL flexible servers
Single Server is retired for both engines, so every open-source estate on Azure now runs on Flexible Server. Azure Data Platform Engineering treats it as the managed engine it is: we still own the query plans, vacuum, connection pooling and version calendar, while Azure owns the host, patching windows and zone-redundant standby.
Zone-redundant HA
In Azure data platform engineering we run a synchronous standby in another availability zone with automatic failover. We measure real failover time from the client side and tune retry logic, because the application sees more than the service promises.
Built-in PgBouncer
For Azure data platform engineering, Flexible Server ships PgBouncer on the server, so connection storms from App Service or AKS do not exhaust backends. We size pool mode and limits from measured concurrency.
Server parameters
shared_buffers, work_mem, autovacuum and max_connections set from the workload, with the change recorded as current value, proposed value, unit and restart requirement.
Read replicas and virtual endpoints
Read scale-out without application rewrites, with replica lag alerting and a promotion runbook tested before it is needed.
Extensions allowlist
pg_stat_statements, pgvector, pg_cron and PostGIS enabled through azure.extensions, checked before migration rather than after.
Version calendar
Major upgrades planned with in-place upgrade or a logical replica, tested on a restored copy, so retirement dates never become incidents.
Deeper engine work runs through our PostgreSQL support and MySQL support teams, with the same Azure data platform engineering on-call covering both. Microsoft publishes platform changes in the Flexible Server release notes, which we track monthly.
Analytics
Microsoft Fabric capacity, OneLake and the Synapse path
Fabric runs every workload against a shared capacity measured in capacity units, and it smooths consumption rather than rejecting spikes immediately. That is convenient until the carried-forward overage catches up and interactive reports start to queue. Azure Data Platform Engineering sizes and schedules Fabric capacity from the Capacity Metrics app, not from the SKU catalogue.
Figure 4. Fabric smoothing and the throttling stages from the Fabric throttling policy; consumption values are illustrative.
For Azure data platform engineering, the key fact is that background operations such as pipeline runs, Spark jobs and dataflow refreshes are smoothed over 24 hours, while interactive queries are smoothed over a much shorter window. A heavy nightly load can therefore still be charged against capacity the next morning. Azure Data Platform Engineering separates capacities by workload where the metrics justify it, moves heavy jobs, and pre-aggregates so Direct Lake semantic models read less.
Under Azure data platform engineering, estates still on Azure Synapse dedicated SQL pools get a planned path: tune distribution and partitioning where the pool stays, or mirror and migrate to the Fabric Warehouse where it does not. Either way, the query ranking alongside is how we find the statements that consume the capacity.
T-SQL · Fabric Warehouse: statement shapes ranked by total elapsed time
-- Fabric Data Warehouse: statement shapes ranked by total elapsed time over 7 days (queryinsights views)
SELECT TOP (20)
query_hash,
COUNT(*) AS executions,
CAST(SUM(total_elapsed_time_ms) / 1000.0 AS DECIMAL(12,1)) AS total_s,
CAST(SUM(allocated_cpu_time_ms) / 1000.0 AS DECIMAL(12,1)) AS cpu_s,
SUM(data_scanned_remote_storage_mb) AS remote_scan_mb,
LEFT(MAX(command), 120) AS sample_statement
FROM queryinsights.exec_requests_history
WHERE start_time >= DATEADD(DAY, -7, GETUTCDATE())
AND status = 'Succeeded'
GROUP BY query_hash
ORDER BY total_s DESC;
Migration
SQL Server to Azure with a gated, reversible cutover
An ageing SQL Server instance nobody dares touch is the most common starting point we see. Azure Data Platform Engineering moves it with replication rather than a weekend outage: the Managed Instance link keeps a readable copy synchronized for weeks, the application team validates against it, and cutover becomes a planned failover behind a written gate.
Figure 5. The five-step migration path through the Managed Instance link, with the cutover gate and the rollback direction.
T-SQL · cutover gate: verify synchronization and empty queues on the source
-- 1. Verify on the SQL Server source: the link's database replicas are synchronized and queues are empty
SELECT ag.name AS availability_group,
ag.is_distributed,
DB_NAME(drs.database_id) AS database_name,
drs.synchronization_state_desc,
drs.log_send_queue_size AS log_send_queue_kb,
drs.redo_queue_size AS redo_queue_kb,
drs.last_commit_time
FROM sys.dm_hadr_database_replica_states AS drs
JOIN sys.availability_groups AS ag ON ag.group_id = drs.group_id
WHERE drs.is_local = 0;
Oracle, MySQL and PostgreSQL sources follow the same principle with Azure Database Migration Service in online mode: full load, change capture, row-count and checksum reconciliation, then a gate. We favour incremental cutovers over big-bang weekends in every Azure data platform engineering plan.
PowerShell · confirmation gate, planned failover and validation
# 2. Confirmation gate, then planned failover of the Managed Instance link (Az.Sql module)
$rg = $env:MI_RESOURCE_GROUP; $mi = $env:MI_NAME; $link = $env:MI_LINK_NAME
$answer = Read-Host "Writes stopped and both queues at 0 for $link? Type the link name to fail over"
if ($answer -ne $link) { Write-Host "Aborted. No change made."; exit 1 }
Start-AzSqlInstanceLinkFailover -ResourceGroupName $rg -InstanceName $mi -Name $link -FailoverType Planned
# 3. Validate: the managed instance is now primary for the link; then repoint connection strings
Get-AzSqlInstanceLink -ResourceGroupName $rg -InstanceName $mi -Name $link | Format-List *
Cost engineering
Where Azure data platform engineering finds Azure data spend
An Azure bill that surprises the CFO almost always traces back to engineering choices. We attribute spend from Cost Management exports to workloads and owners, then fix the causes and verify the savings with the same query a month later.
| Service | Where spend hides | What we change | Evidence |
|---|---|---|---|
| Azure SQL Database | vCores sized for a rare peak; Business Critical where latency does not need it | Right-size from measured load; serverless with auto-pause for intermittent databases; reservations for the steady baseline | sys.dm_db_resource_stats, Cost Management |
| Managed Instance | Idle development instances; licence not using Azure Hybrid Benefit | Stop/start schedules where supported; hybrid benefit where licences allow | Instance utilisation, licence inventory |
| Cosmos DB | Provisioned RU/s far above consumed; cross-partition queries | Autoscale or serverless; key redesign; indexing policy trimmed | Normalized RU consumption, diagnostic logs |
| Microsoft Fabric | Capacity sized for a twice-yearly peak; one capacity for every workload | Split capacities; reschedule background jobs; pause non-production | Capacity Metrics app |
| Synapse dedicated pools | Pools online over weekends; replicated tables that should be hash-distributed | Pause schedules; distribution fixes; Fabric migration plan | DMV skew and data movement steps |
| Storage and network | Premium disks and geo-replication where not required; egress between regions | Tier and replication review; co-locate compute and data | Cost by meter and resource |
Beyond Azure data platform engineering reviews, programme-level cost work runs under our cloud database optimization and FinOps practice, alongside the AWS and Google Cloud data platform engineering teams for multi-cloud estates.
Governance and security
Governance auditors and engineers both accept
In Azure data platform engineering, security teams want provable control; engineers want to ship without friction. The Azure governance stack, assembled well, gives both, and Azure data platform engineering maps each obligation to an enforced control rather than a policy document.
| Obligation | Azure control | How we prove it |
|---|---|---|
| Identity and least privilege | Microsoft Entra ID authentication, PIM for admin roles, managed identities, contained users | Role assignment inventory, PIM activation logs |
| Network isolation | Private endpoints, public access denied by Azure Policy | Resource Graph query, policy compliance state |
| Encryption | TDE with customer-managed keys in Key Vault; TLS 1.2+ enforced | Key Vault audit, server TLS settings |
| Data protection in the engine | Row-level security, dynamic data masking, Always Encrypted where required | Policy and masking inventory per database |
| Catalog and lineage | Microsoft Purview scans, classification, lineage from pipelines | Classified-asset reports, lineage for audited reports |
| Audit and threat detection | Azure SQL auditing to Log Analytics, Defender for SQL | KQL queries answering who read what, and alerts triaged |
| Regulation and residency | Region policy, geo-replication scoping; SOC 2, ISO 27001, HIPAA, GDPR, India’s DPDP Act | Control-to-evidence map per framework |
Test every tier change, parameter change, policy assignment and failover in a non-production subscription first, keep point-in-time restore and geo-backups verified by restore drills, and maintain a robust disaster-recovery posture with a tested secondary region.
Engagement model
From assessment to 24×7 operations
We meet the estate where it is: a greenfield Fabric build, a stalled SQL Server migration, or a platform that works but costs too much. Every Azure data platform engineering engagement starts with two weeks of measurement, because evidence beats a quarter of strategy decks.
Phase 1 · weeks 1–2
Assess
The Azure data platform engineering assessment covers landing-zone gaps, tier fit, Cosmos DB keys, Fabric capacity, spend and risks, ranked and costed.
Phase 2
Architect
Target estate, tier and capacity decisions, network and identity design, migration waves and rollback criteria.
Phase 3
Engineer
Build, migrate and tune with senior engineers, validated at every cutover gate.
Phase 4
Operate
24×7 operations under SLA, monthly cost and performance reviews, quarterly DR drills.
Related services: remote DBA services, data analytics and data warehousing support and 24/7 emergency DBA coverage.
FAQ
Questions about Azure Data Platform Engineering with MinervaDB
The questions data leaders ask most often before an Azure data platform engineering engagement.
Which Azure SQL tier should we use?
It depends on measured I/O latency, log rate, failover tolerance and growth. General Purpose suits cost-sensitive, latency-tolerant work; Business Critical suits OLTP that needs the lowest latency; Hyperscale suits large or fast-growing databases and read scale-out.
Azure SQL Database or Managed Instance?
Azure SQL Database for new, database-scoped applications; Managed Instance when you need instance-level SQL Server features such as Agent jobs, cross-database queries or CLR, or a near-lift-and-shift migration through the Managed Instance link.
How do you migrate SQL Server without a big-bang weekend?
With the Managed Instance link or online Azure Database Migration Service: replicate for weeks, validate on the target, then cut over behind a gate that requires empty send and redo queues, a typed confirmation and a prepared rollback path.
Why is our Cosmos DB throttling when average RU usage is low?
It is the most common Azure data platform engineering finding on Cosmos DB. Provisioned throughput is divided across physical partitions, so a hot partition key hits its share while others sit idle. We find the key in diagnostic logs and redesign it with higher cardinality, hierarchical or synthetic keys.
Why do Fabric reports slow down in the morning?
Background jobs are smoothed over 24 hours, so a heavy overnight load can push the capacity into carry-forward overage and delay interactive work. We reschedule, split capacities and pre-aggregate based on the Capacity Metrics app.
Should we stay on Synapse dedicated pools or move to Fabric?
It depends on workload, feature use and cost. We tune pools that should stay and plan mirroring and migration to the Fabric Warehouse where it pays off, with measured before-and-after numbers.
Do you only consult, or do you also run the platform?
Both. Most Azure data platform engineering engagements continue into 24×7 managed operations with a 15-minute Severity-1 response, monthly reviews and quarterly disaster-recovery drills.
Will you tell us when Azure is the wrong answer?
Yes. We are vendor-neutral and do not resell Azure. When a workload is cheaper or safer self-managed, on another cloud or on another engine, we say so in writing before any project starts.
Build an Azure data estate that performs and pays off
Start with a two-week assessment of landing zone, tiers, partitioning, capacity and spend, ranked by impact. Azure Data Platform Engineering for active incidents is available through the same page, 24×7.