S3 - AWS Data Engineering

S3 Features — 10 Fundamentals

S3 (Simple Storage Service) is scalable, durable (11 9s), secure, and highly available. From the lessons.txt feature list:

Scalable storage — stores structured, semi-structured, unstructured data
Durability 99.999999999% — 11 nines
Highly protected & replicated across regions — avoids redundancy loss
Security: Encryption (SSE-S3 AES256, SSE-KMS, DSSE-KMS, client-side) + IAM/Bucket Policies + ACLs
Lifecycle management — auto-transition between storage classes
Event notifications (SNS) on create/delete/change
Cross-Region Replication (CRR) — disaster recovery
Logging & Monitoring via CloudWatch — every action recorded
MFA support
Access control — IAM, bucket policies (JSON), ACLs (legacy)

Interview Tip

Emphasize durability, replication, and that S3 is not just storage but an event source for Lambda/Glue.

S3: 11 nines durability, cross-region replication

S3 Storage Classes

Class	Access	Retrieval	Cost
S3 Standard	Frequent	Millis, always available	Higher
Intelligent-Tiering	Auto frequent ↔ infrequent	Millis	Optimized, cheaper than Standard
S3 Glacier	Archival	Minutes to 12 hours	Very low
Glacier Deep Archive	Rarely accessed	~12 hours	Lowest
One Zone-IA	Infrequent, single AZ	Millis	Low

Choose Standard for hot data, Intelligent-Tiering for unknown patterns, Glacier for archival.

S3 Security — Encryption & Access Control

Encryption

Server-Side (SSE-S3 AES256) — AWS encrypts after receipt, manages keys
SSE-KMS — AWS KMS manages keys, auditable
DSSE-KMS — dual-layer with KMS
Client-Side — encrypt on client before send (e.g., WhatsApp), client manages keys
In-Transit — TLS during transfer

Bucket Policy Example (from lessons.txt)

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": "*",
    "Action": "s3:GetObject",
    "Resource": "arn:aws:s3:::mybucket/*"
  }]
}

IAM vs ACL

IAM Policies — centralized, user/group/role (RBAC), bucket-level
ACLs — object/bucket level, legacy, AWS recommends IAM

Versioning, Delete & Logging

Enable versioning — every upload gets a unique VersionId. S3 never overwrites, it creates a new version.

mydata.csv v1.1, mydata1.csv — each has distinct VersionId
Delete in versioned bucket adds a delete marker, not removal — delete the marker to restore, or specify VersionId to permanently delete
Logging/Monitoring via CloudWatch (application/system/access/security logs), CloudWatch Events/EventBridge for notifications

Folder Structure — E-commerce Example (from lessons.txt)

From the prompt: 10 CSV via SFTP + 2 REST APIs (Google Analytics, Facebook) + 15 JDBC tables. Organize by zone and source with date prefix:

s3://your-bucket-name/
├── raw_data/
│   ├── FINANCE/csv_files/2025-03-07_file1.csv
│   ├── SALES/google_analytics/2025-03-07_google_analytics_data.json
│   ├── SALES/facebook_insights/2025-03-07_facebook_insights_data.json
│   └── SALES/jdbc_tables/table1/2025-03-07_table1_part1.csv
├── processed_data/cleaned/2025-03-07_cleaned_file1.csv
├── staging_data/intermediate/
├── archive/2025-03/
└── metadata/sftp_file_manifest.json

Code from file:

filename = 'finance' + datetime.now().strftime('%d%m%y') + '.csv'
link = 'c://myfolder/finance/csv/' + filename

Tip

Use yyyy-mm-dd prefix and separate raw/processed/staging/archive to make data easily accessible and lifecycle-manageable.

Cross-Account Sharing (Account A → B)

Account A adds bucket policy trusting Account B
Account B creates access point linked to Account A bucket
Account B adds access point policy granting IAM roles
Account B attaches IAM policy to the role
Account B uses access point ARN instead of bucket name

Bookmarks