Simple ETL

This guide shows you how to build a file processing pipeline using AWS S3, Lambda, and DynamoDB locally with LocalStack. This lab demonstrates how S3 uploads trigger a Lambda to log metadata into a DynamoDB table.

Prerequisites

• Docker installed and running.
• Dev Containers extension installed (optional).
• AWS Toolkit for VS Code (optional, included in Dev Container).

How to execute

1. Setup Environment:

   Dev Container (recommended)

   Open VS Code in the project folder and execute this command in the Command Palette:

   > Dev Containers: Reopen in Container

   Manually
   1. Start LocalStack:

      docker compose up -d localstack

   2. Configure AWS CLI:

      aws configure set aws_access_key_id test --profile localstack
      aws configure set aws_secret_access_key test --profile localstack
      aws configure set region us-east-1 --profile localstack
      aws configure set output json --profile localstack
      aws configure set endpoint_url http://localhost:4566 --profile localstack
      aws configure set cli_pager "" --profile localstack

   3. Install Python dependencies:

      pip install uv
      uv sync

   4. Package the Lambda function:

      python deploy/utils/package_lambda.py

   5. Deploy:

      python deploy/boto3_deploy.py

2. Run the Example:

   Python

   python main.py

   cURL

   curl -X PUT -T src/lambda.py http://localhost:4566/file-uploads-bucket/manual-upload.py

   AWS Toolkit
   1. Select the localstack profile in the AWS Toolkit.
   2. Right-click on the file-uploads-bucket and select Upload Files….

Deployment methods

There are other deployment methods that you can use in this Lab.

Terraform

terraform -chdir=deploy init
terraform -chdir=deploy apply -auto-approve

CloudFormation

1. Create a temporary bucket for deployment:

   aws s3 mb s3://lambda-deploy-bucket --profile localstack

2. Upload the Lambda package:

   aws s3 cp tmp/lambda.zip s3://lambda-deploy-bucket/lambda.zip --profile localstack

3. Deploy the stack:

   aws cloudformation deploy --profile localstack \
      --stack-name aws-dynamo-db-stack \
      --template-file deploy/cloud_formation_deploy.yaml \
      --capabilities CAPABILITY_NAMED_IAM

Boto3

python deploy/boto3_deploy.py

AWS CLI

1. Create DynamoDB Table:

   aws dynamodb create-table --profile localstack \
     --table-name file-logs \
     --attribute-definitions AttributeName=file_id,AttributeType=S \
     --key-schema AttributeName=file_id,KeyType=HASH \
     --billing-mode PAY_PER_REQUEST

2. Create IAM Role:

   aws iam create-role --profile localstack \
     --role-name lambda-s3-processor-role \
     --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"lambda.amazonaws.com"},"Action":"sts:AssumeRole"}]}'

3. Create Lambda Function:

   aws lambda create-function --profile localstack \
     --function-name s3-file-processor \
     --runtime python3.12 \
     --role arn:aws:iam::000000000000:role/lambda-s3-processor-role \
     --handler lambda.lambda_handler \
     --zip-file fileb://tmp/lambda.zip \
     --environment Variables={DYNAMODB_TABLE=file-logs}

4. Create S3 Bucket:

   aws s3 mb s3://file-uploads-bucket --profile localstack

5. Add S3 Permission and Configure Notification:

   aws lambda add-permission --profile localstack \
     --function-name s3-file-processor \
     --statement-id s3-trigger \
     --action lambda:InvokeFunction \
     --principal s3.amazonaws.com \
     --source-arn arn:aws:s3:::file-uploads-bucket
   aws s3api put-bucket-notification-configuration --profile localstack \
     --bucket file-uploads-bucket \
     --notification-configuration '{"LambdaFunctionConfigurations":[{"LambdaFunctionArn":"arn:aws:lambda:us-east-1:000000000000:function:s3-file-processor","Events":["s3:ObjectCreated:*"]}]}'

Switching deployment methods

If you switch between Terraform, CloudFormation, Boto3, or AWS CLI, run Clean Up first to avoid resource name conflicts.

Validate results

AWS CLI

• Check S3 Bucket:

  aws s3 ls s3://file-uploads-bucket --profile localstack

• Scan DynamoDB Table:

  aws dynamodb scan --table-name file-logs --profile localstack

• View Lambda Logs:

  aws logs tail /aws/lambda/s3-file-processor --profile localstack

AWS Toolkit

• S3: Expand the S3 section to see the uploaded files.
• CloudWatch: Expand the Logs section to see the Lambda execution output.

NoSQL Workbench

1. Download: NoSQL Workbench for DynamoDB.
2. Setup Connection: Click Operation builder > Add connection > DynamoDB local. Set Connection Name to LocalStack, Hostname to localhost, Port to 4566.
3. Connect to browse and edit the file-logs table.

Clean Up

docker compose down -v

Troubleshooting

Issue	Solution
Connection Refused	Ensure LocalStack is running and wait for the Ready. message in logs.
Lambda Not Triggering	Verify logs: aws logs tail /aws/lambda/s3-file-processor --profile localstack

Next steps

• See the Google Cloud equivalent project.
• Dive into the Lambda (SAM) Lab.
• Dive into the S3 (MinIO) Lab.