EMR - Elastic MapReduce
EMR is an AWS managed cluster platform that simplifies running big data frameworks like Apache Spark and Hadoop on AWS to process and analyze vast amounts of data.
Processes large amounts of raw data: logs, transaction data, clickstream (Google Analytics).
EMR: Spark on a cluster to transform data. Uses EC2 instances.
We decide how many nodes, what type of nodes, and when to scale up or down.
More control on the nodes.EMR Architecture
Master Node (Team Leader)
The leader node. Keeps track of progress, makes sure everything is running smoothly. If the master node fails, the entire project fails. Schedules, monitors, and assigns tasks.
Core Nodes (Team Members)
They store parts of data and process that data by running tasks. If a core node fails, we might lose some data.
Task Nodes (Support Workers)
Optional nodes which only support core nodes to process data but they don't store data. You can add them when you need extra processing power. If a task node fails, no data is lost.
Storage Options
| Storage | Description | Persistence |
|---|---|---|
| HDFS | Hadoop Distributed File System. Like storing files across multiple team members. Fast but temporary. | Lost when cluster shuts down |
| Amazon S3 | Permanent library. Data remains safely even after cluster is done. | Persistent |
| EMRFS | EMR File System. Like native memory, faster to read/write to S3. | Persistent (S3-backed) |
| Local File System | Individual memory in each core node. For temporary calculations. | Lost on node termination |
How EMR Works
EMR uses distributed processing techniques:
- Hadoop MapReduce - Break the big problem into smaller ones: Map (split data and perform initial processing) and Reduce (combine results into a final answer).
- Apache Spark - Faster alternative that keeps data in memory rather than writing to disk after each step.
Example: Count words in millions of docs
1) The map could count words in each document
2) The reduce would add up all the counts for each unique word
End:
- Results are saved to S3
- Keep the cluster running for more work or shut it down to save money
- If you shut it down, you only pay for the time you usedEMR Code Example
Transform log files in S3 to find hourly product views:
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, hour
spark = SparkSession.builder.appName('EMR').getOrCreate()
logs = spark.read.csv('s3://bucket/logs.csv', header=True)
dfclean = logs.dropna().filter('age > 18')
dfclean.write.parquet('s3://output/path')EMR vs Glue
| Feature | EMR | Glue |
|---|---|---|
| Infrastructure | Cluster based (EC2 instances) | Serverless |
| Engines | Spark and Hadoop | Spark and Python |
| Use Case | Complex transformations, heavy analytics | Simple transformations, occasional ETL |
| Flexibility | More flexible, full control | Less flexible, managed service |
| Cost | Pay for EC2 + EMR (even idle). Cheaper for heavy workloads. | Pay per second (only when running). Cheaper for small/medium ETL. |
| Expertise | More technical expertise needed | Less technical expertise needed |
| Scalability | More scalable (manual cluster management) | Less scalable (auto-scaling within limits) |