AWS Glue — Serverless ETL

Glue is serverless (no backend to maintain), scales automatically, supports Python/Scala/PySpark. Versions from file: Glue 4, Python 3.11-3.13, Spark 3.3, Pandas 1.3.

Sources: S3 (CSV/JSON/Parquet/Avro), JDBC (RDS/Redshift/MySQL/Postgres/SQL Server), DynamoDB, Kinesis, REST API, SFTP. Classifiers detect JSON/CSV/Parquet.

Components

Glue can import data from multiple sources:

1. AWS S3 — Data Lake (CSV, JSON, Parquet, Avro, etc.)
2. JDBC connections — AWS RDS, Redshift, MySQL, PostgreSQL, SQL Server
3. Amazon DynamoDB — NoSQL database
4. AWS Kinesis Data Streams — real-time data source
5. REST API, SFTP

Classifiers: JSON, CSV, Parquet, etc. — Glue uses classifiers to detect file formats and infer schemas during crawling.

AWS Data Catalog: Centralised metadata repository / centralised metadata repository — metadata store that holds all definitions of data sources, tables and data schemas. Metadata = data about data (datatype, size, format).

Automatic Schema Discovery: AWS Glue can automatically discover and catalog data schemas from various data sources via Crawlers. Crawlers scan the data source to infer schema and create and update tables in the Glue Data Catalog. Schema Evolution: crawlers automatically update the catalog when source schema changes. Crawlers can be scheduled at regular intervals, ensuring metadata is always up to date.

AWS Glue Studio: drag and drop tool to create ETL jobs visually.

AWS Glue ETL Jobs (Glue Jobs): can be written in Python, Scala, PySpark — extract from source, transform data using transformation logic (filtering, joining, deduplication, fillna, dropna, joins, aggregated columns), load to final destination (S3, Redshift, PostgreSQL, MySQL).

Data Catalog, Crawlers & Schema Registry

Data Catalog = centralized metadata repository (data about data: types, size, format). Crawlers scan sources to infer schema and create/update tables (schema evolution). Schema Registry stores schema versions with compatibility modes:

Crawler Triggers

Scheduled (12am daily → 12:30am crawler, cron 30 0 * * ? *) — simple but not real-time, may run even if no data.

Event-based (S3 PUT → Lambda → start_crawler) — real-time, only runs on upload.

import boto3
def lambda_handler(event,context):
    glue = boto3.client('glue')
    glue.start_crawler(Name='name of the crawler')
    return {'statusCode':200,'body':'Crawler started'}

S3 → Event Notification (PUT on bucket) → Lambda → EventBridge (crawler Succeeded) → Glue Job. Alternative: Glue Workflows (crawler → job → notification).

DynamicFrames vs DataFrames

From file: DynamicFrames are flexible, DataFrames are strict.

Example from file: order 103 has '300' (string) and 104 has 'na' — DataFrame fails, DynamicFrame creates choice type int or string and you resolve:

Python Copy

dyf = dyf.resolveChoice(specs=[('amount', 'cast:int')])
df = dyf.toDF()
# Convert back
# df to dyf: dyf = DynamicFrame.fromDF(df, glueContext, 'mydynamicframe')

Flow: raw → Glue reads DynamicFrame → resolve choice → transform → convert to DataFrame → write to S3.

Also auto-merges schemas: file1 (orderid, amount) + file2 (orderid, amount, discount) → single schema.

DPU & Worker Types

1 DPU = 4 vCPU + 16 GB RAM. Number of workers × DPU per worker = total DPUs.

Example from file: 500GB daily, heavy joins → G.2X, 20 workers, 40 DPUs, auto-scaling true.

Glue Crawler Details

Crawlers infer schema and update Data Catalog, handling schema evolution. Scheduled vs event-based as above.

Give S3 permission to invoke Lambda, and Lambda permission to start crawler via IAM.

Glue ETL Jobs

Written in Python/Scala/PySpark: extract → transform (filter, join, deduplication) → load to S3/Redshift/Postgres/MySQL.

from awsglue.context import GlueContext
from pyspark.context import SparkContext
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session

dyf = glueContext.create_dynamic_frame.from_catalog(database='landingdb', table_name='sales')
newdyf = dyf.dropDuplicates()
df = newdyf.toDF()
df.write.parquet('s3://finals3bucket', index=False)

Flow: S3 PUT → Lambda → Crawler (EventBridge on Succeeded → Glue Job) or Workflows.

Workflows, Bookmarks & CDC

Workflows: Glue Workflows orchestrate crawler → job → notification. Alternative to EventBridge.

Job Bookmarks: Track last processed record (hidden table in Data Catalog, often via timestamp). Enable in Job parameters → enable job bookmarks → incremental loads.

CDC via AWS DMS: full load + CDC, captures inserts/updates/deletes via timestamp columns (created_at, updated_at), avoids full scan, near real-time, lands to S3 then crawler → Redshift.

Other incremental methods from file: file timestamp & naming convention (mydata_01012025.json), Lambda on S3, prefix/suffix filtering.