AWS Data Platform Engineering, Migration & Managed Operations

Schema and code conversion is where timelines slip, so it starts first. DMS Schema Conversion handles the mechanical share; PL/SQL packages, T-SQL procedures and vendor-specific data types get engineer review in Git. Large tables load in parallel segments, indexes are built after the load, and CDC then runs for weeks while the application team tests against the target.

Before cutover every table must show ValidationState = Validated under DMS data validation, CDC latency must hold under the agreed threshold, and a reverse replication task must already exist so rollback is a switch rather than a restore. The script alongside is the gate our AWS data platform engineering team runs: read-only checks, a typed confirmation, then validation after the change.

#!/usr/bin/env bash
# DMS cutover gate: read-only checks first, then an explicit confirmation before forward replication stops.
set -euo pipefail
TASK_ARN=${DMS_TASK_ARN:?set DMS_TASK_ARN}
TASK_ID=${DMS_TASK_ID:?set DMS_TASK_ID}; INSTANCE_ID=${DMS_INSTANCE_ID:?set DMS_INSTANCE_ID}

# 1. Verify: every table validated, no mismatched records
aws dms describe-table-statistics --replication-task-arn "$TASK_ARN" \
  --query "TableStatistics[?ValidationState!='Validated'].[SchemaName,TableName,ValidationState,ValidationFailedRecords]" \
  --output table

# 2. Verify: CDC latency on the target over the last 15 minutes (seconds)
aws cloudwatch get-metric-statistics --namespace AWS/DMS --metric-name CDCLatencyTarget \
  --dimensions Name=ReplicationInstanceIdentifier,Value="$INSTANCE_ID" Name=ReplicationTaskIdentifier,Value="$TASK_ID" \
  --start-time "$(date -u -d '-15 min' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
  --period 60 --statistics Maximum --query "sort_by(Datapoints,&Timestamp)[-5:].[Timestamp,Maximum]" --output table

# 3. Confirmation gate: application writes must already be frozen at the source
read -r -p "Writes frozen, all tables Validated, latency at 0? Type CUTOVER to stop forward CDC: " ANSWER
[[ "$ANSWER" == "CUTOVER" ]] || { echo "aborted, replication still running"; exit 1; }

aws dms stop-replication-task --replication-task-arn "$TASK_ARN"

# 4. Validate: task stopped cleanly; start the pre-built reverse task that keeps the old source as rollback target
aws dms describe-replication-tasks --filters Name=replication-task-arn,Values="$TASK_ARN" \
  --query "ReplicationTasks[0].[Status,StopReason]" --output text

Iceberg’s hidden partitioning means queries filter on event_ts and the engine prunes partitions without users knowing the layout. Manifest statistics add file-level pruning on top, and columnar Parquet with ZSTD reads only the columns a query touches. Streaming writes create many small files, so compaction and snapshot expiry run on a schedule, per the Athena Iceberg documentation.

Where teams prefer a managed table store, Amazon S3 Tables provide Iceberg tables with automatic maintenance, and Glue zero-ETL can land DynamoDB and SaaS data directly into them. AWS Data Platform Engineering chooses between self-managed Iceberg and S3 Tables on cost and control, and catalogues both in Glue with Lake Formation permissions.

-- Athena (engine v3): curated Iceberg table with hidden partitioning and ZSTD Parquet
CREATE TABLE lake_curated.events (
    event_id     STRING,
    customer_id  BIGINT,
    event_type   STRING,
    amount       DECIMAL(18,2),
    event_ts     TIMESTAMP
)
PARTITIONED BY (day(event_ts), bucket(16, customer_id))
LOCATION 's3://${LAKE_BUCKET}/curated/events/'
TBLPROPERTIES (
    'table_type'        = 'ICEBERG',
    'format'            = 'parquet',
    'write_compression' = 'zstd'
);

-- Weekly maintenance: compact small files from streaming writes, then expire old snapshots
OPTIMIZE lake_curated.events REWRITE DATA USING BIN_PACK
WHERE event_ts >= current_timestamp - INTERVAL '7' DAY;

VACUUM lake_curated.events;   -- honours vacuum_max_snapshot_age_seconds; time travel before it is gone