Lambda - AWS Data Engineering

Lambda — Serverless Compute

Serverless, no servers to manage, auto-scales, event-driven (S3, DynamoDB, Glue, CloudWatch) or time-based, stateless, supports Python/Node/Java/C#/Go. From file: mainly used for triggers.

Lambda Layers

Lambda Layers: Predefined packages (e.g., pandas layers) or custom packages via AWS CLI to use within Lambda.

Limitations

Limit	Value
Timeout	15 minutes
Memory	10 GB
Deployment package	50 MB (250 MB via S3)
Concurrency	1000 per region (increasable)
Cold start	Latency on first invoke

Boto3 — Invoke Lambda

import boto3
lambda_client = boto3.client('s3', region_name='us-east-1')
response = lambda_client.invoke(
  FunctionName='my_lambda_function',
  InvocationType='RequestResponse'
)
print(response['Payload'].read().decode())

Invoke Glue Job via Lambda

import boto3, json
def lambda_handler(event, context):
    # event has S3 Records: bucket name and key
    bucketname = event['Records'][0]['s3']['bucket']['name']
    key = event['Records'][0]['s3']['object']['key']
    glue_client = boto3.client('glue', region_name='us-east-1')
    response = glue_client.start_job_run(
        JobName='my_glue_job',
        Arguments={'--arg1':'value1'}
    )
    return {'statusCode':200,'body': json.dumps('Glue job started')}

Event contains Records[0].s3.bucket.name and object.key. Context has function name, memory, remaining time, request ID.

Invoke Glue Crawler via Lambda

Use boto3 to trigger a Glue crawler from a Lambda function. This is event-driven — when data lands in S3, the Lambda starts the crawler automatically.

import json
import boto3

glue_client = boto3.client('glue')

def lambda_handler(event, context):
    crawler_name = 'landingbucket-crawler'
    print('event received: ', json.dumps(event))

    try:
        response = glue_client.start_crawler(Name=crawler_name)
        print('Crawler started successfully')
        return {
            'statusCode': 200,
            'body': json.dumps('Hello from Lambda!')
        }
    except glue_client.exceptions.CrawlerRunningException:
        print('Crawler is already running')
        return {
            'statusCode': 200,
            'body': json.dumps('Crawler is already running')
        }
    except Exception as e:
        print('Error starting crawler: ', str(e))
        return {
            'statusCode': 500,
            'body': json.dumps('Error starting crawler')
        }