Skip to content
MistLock MistLock
Star
478
Star
478

S3 (Garage)

This guide shows you how to work with AWS S3 using Garage as a local emulator. This lab demonstrates a data pipeline using Boto3, PyArrow, and Delta Lake (delta-rs).

Prerequisites

Section titled “Prerequisites”

How to execute

Section titled “How to execute”

  1. Setup Environment:

    Open VS Code in the project folder and execute this command in the Command Palette:

    Terminal window
    >  Dev Containers: Reopen in Container

  2. Run the Example:

    Terminal window
    python main.py

How to debug

Section titled “How to debug”

  1. main.py:

    • Open: Open main.py.
    • Breakpoints: Set breakpoints in the code.
    • Run: Press F5 to start debugging.

  2. Tests:

    • Open: Open a test file (e.g., tests/components/test_s3_boto.py).
    • Breakpoints: Set breakpoints in the test code.
    • Run: Use the VS Code Testing tab and click the Debug Test icon next to the test you want to debug.

How to test

Section titled “How to test”
  • Individually: Run tests individually from the VS Code Testing tab.
  • All tests: Run the automated script:
    Terminal window
    scripts/run_tests.sh

Validate results

Section titled “Validate results”
  • Check Buckets: Verify that bronze and silver buckets were created.
    Terminal window
    aws s3 ls --profile garage
  • Check Content: Verify the files in the silver bucket.
    Terminal window
    aws s3 ls s3://silver --recursive --profile garage

Clean Up

Section titled “Clean Up”
Terminal window
docker compose down -v

Troubleshooting

Section titled “Troubleshooting”
IssueSolution
Garage API not readyEnsure the garage container is running and wait a few seconds for the API to initialize. Check logs with docker logs garage.
Port 3900 in useStop any other service using port 3900 or change the mapping in docker-compose.yml.

Next steps

Section titled “Next steps”