Data & Analytics Skills

Build a real-time change-data-capture pipeline from a transactional database (Aurora DSQL, Aurora Postgres, RDS) to Redshift Serverless, S3, or any other sink, via Kinesis Data Streams (or MSK). Covers producer setup, consumer Lambda correctness, the append-only event log + reconstruction pattern, schema-drift handling with SUPER + JSONPARSE, and the operational gotchas (Redshift Data API parameter caps, async statement polling, poison-record handling, retry-and-bisect). Use when the user asks for "CDC to Redshift", "stream change data capture", "Aurora CDC", "DSQL CDC", "Postgres replication to S3", or "build a Lambda consumer for Kinesis CDC". Aurora DSQL public-preview specifics are flagged; the rest is source-agnostic.

Build a real-time analytics pipeline downstream of Aurora DSQL using the DSQL CDC stream. Covers consumer Lambda correctness, the append-only event log + ROWNUMBER reconstruction pattern that absorbs unordered/duplicate delivery, Redshift Data API parameter-cap chunking, async statement polling, BisectBatchOnFunctionError, MaximumRecordAgeInSeconds, DLQ wiring, and the DSQL-preview gotcha that both INSERT and UPDATE arrive as op='c'. Triggers on phrases like: DSQL to analytics, DSQL to Redshift, analyze DSQL data, real-time DSQL analytics, DSQL CDC pipeline, Aurora DSQL change data capture, stream DSQL changes, replicate DSQL, DSQL to S3, DSQL to Iceberg, DSQL Kinesis Lambda, DSQL data warehouse.

Build an Amazon Data Firehose delivery stream that writes to Apache Iceberg tables on S3 Tables via IcebergDestinationConfiguration. Covers the column-shape footgun (records land in the error bucket with Iceberg.MissingColumnWithinRecord unless a transform Lambda reshapes them), the microsecond timestamp requirement, the three-phase deploy pattern that works around Firehose's synchronous glue:GetTable validation at create time, parameter-inheritance traps in aws cloudformation deploy, Lake Formation grant ordering, and how to decode the error bucket. Use when you see "Firehose to Iceberg", "Firehose to S3 Tables", "IcebergDestinationConfiguration", "Firehose data transformation Lambda", "Firehose error bucket", "Iceberg.MissingColumnWithinRecord", "MissingColumnWithinRecord", or any time someone is wiring a Kinesis Data Stream / Direct PUT source to Iceberg tables and producer record shape does not match the Iceberg column layout.

Build a lakehouse on AWS that Redshift Serverless can query - S3 Tables (managed Apache Iceberg) for storage, the Glue federated catalog (s3tablescatalog) for metadata, Lake Formation for access control, and a Redshift Serverless external schema for SQL. Use when the user wants to "build a lakehouse on AWS", "set up a data lake on AWS", "query Iceberg from Redshift Serverless", "wire S3 Tables to Redshift", "set up the Glue federated catalog for S3 Tables", or "let Redshift Serverless read S3-resident Iceberg tables". Covers the highest time-wasted gotchas in this stack: !Ref AWS::S3Tables::TableBucket returning the ARN (not the name), the bucket-nested Glue catalog ARN format, the " @s3tablescatalog" 3-part naming, IAM access control mode versus Lake Formation access control mode, and which Redshift external-schema path to pick. Does NOT cover Firehose-specific authoring (use firehose-iceberg-pipeline) or hot-path Kinesis/Lambda CDC ingestion (use cdc-streaming-pipeline).

Stream rows continuously into Apache Iceberg tables on S3 Tables (or standard Iceberg on a general purpose bucket) using Amazon Data Firehose with IcebergDestinationConfiguration. Covers the Firehose stream, the optional record-shaping Lambda (ProcessingConfiguration), the Lake Formation grants required for Firehose to write through Glue, and the three-phase CloudFormation deploy that gates the stream on grants existing. Triggers on: stream into data lake, Firehose to S3 Tables, Firehose to Iceberg, real-time ingestion to Iceberg, Firehose IcebergDestinationConfiguration, stream Kinesis to Iceberg, near real-time CDC to data lake, continuous append to Iceberg. Do NOT use for: batch file imports (use ingesting-into-data-lake), creating the destination Iceberg table itself (use creating-data-lake-table), the one-time Lake Formation onboarding (use setting-up-lake-formation), finding tables by fuzzy name (use finding-data-lake-assets), running queries (use querying-data-lake), Glue connections (Firehose does not use them; do not invoke connecting-to-data-source), Kinesis to Lambda to Redshift Data API direct writes (not Iceberg; recommend a different workflow), or producer-side Kafka or MSK setup.