AWS Glue — Serverless ETL

Glue is serverless (no backend to maintain), scales automatically, supports Python/Scala/PySpark. Versions from file: Glue 4, Python 3.11-3.13, Spark 3.3, Pandas 1.3.

Sources: S3 (CSV/JSON/Parquet/Avro), JDBC (RDS/Redshift/MySQL/Postgres/SQL Server), DynamoDB, Kinesis, REST API, SFTP. Classifiers detect JSON/CSV/Parquet.

Data Catalog, Crawlers & Schema Registry

Data Catalog = centralized metadata repository (data about data: types, size, format). Crawlers scan sources to infer schema and create/update tables (schema evolution). Schema Registry stores schema versions with compatibility modes:

Crawler Triggers

Scheduled (12am daily → 12:30am crawler, cron 30 0 * * ? *) — simple but not real-time, may run even if no data.

Event-based (S3 PUT → Lambda → start_crawler) — real-time, only runs on upload.

import boto3
def lambda_handler(event,context):
    glue = boto3.client('glue')
    glue.start_crawler(Name='name of the crawler')
    return {'statusCode':200,'body':'Crawler started'}

S3 → Event Notification (PUT on bucket) → Lambda → EventBridge (crawler Succeeded) → Glue Job. Alternative: Glue Workflows (crawler → job → notification).

DynamicFrames vs DataFrames

From file: DynamicFrames are flexible, DataFrames are strict.

Example from file: order 103 has '300' (string) and 104 has 'na' — DataFrame fails, DynamicFrame creates choice type int or string and you resolve:

Python Copy

dyf = dyf.resolveChoice(specs=[('amount', 'cast:int')])
df = dyf.toDF()
# Convert back
# df to dyf: dyf = DynamicFrame.fromDF(df, glueContext, 'mydynamicframe')

Flow: raw → Glue reads DynamicFrame → resolve choice → transform → convert to DataFrame → write to S3.

Also auto-merges schemas: file1 (orderid, amount) + file2 (orderid, amount, discount) → single schema.

DPU & Worker Types

1 DPU = 4 vCPU + 16 GB RAM. Number of workers × DPU per worker = total DPUs.

Example from file: 500GB daily, heavy joins → G.2X, 20 workers, 40 DPUs, auto-scaling true.

Glue Crawler Details

Crawlers infer schema and update Data Catalog, handling schema evolution. Scheduled vs event-based as above.

Give S3 permission to invoke Lambda, and Lambda permission to start crawler via IAM.

Glue ETL Jobs

Written in Python/Scala/PySpark: extract → transform (filter, join, deduplication) → load to S3/Redshift/Postgres/MySQL.

from awsglue.context import GlueContext
from pyspark.context import SparkContext
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session

dyf = glueContext.create_dynamic_frame.from_catalog(database='landingdb', table_name='sales')
newdyf = dyf.dropDuplicates()
df = newdyf.toDF()
df.write.parquet('s3://finals3bucket', index=False)

Flow: S3 PUT → Lambda → Crawler (EventBridge on Succeeded → Glue Job) or Workflows.

Workflows, Bookmarks & CDC

Workflows: Glue Workflows orchestrate crawler → job → notification. Alternative to EventBridge.

Job Bookmarks: Track last processed record (hidden table in Data Catalog, often via timestamp). Enable in Job parameters → enable job bookmarks → incremental loads.

CDC via AWS DMS: full load + CDC, captures inserts/updates/deletes via timestamp columns (created_at, updated_at), avoids full scan, near real-time, lands to S3 then crawler → Redshift.

Other incremental methods from file: file timestamp & naming convention (mydata_01012025.json), Lambda on S3, prefix/suffix filtering.