S3 (MinIO)

This guide shows you how to work with AWS S3 locally using MinIO and Python. This lab demonstrates how to integrate S3 libraries (Boto3, PyArrow, and Deltalake) for local data pipelines and object storage emulation.

Prerequisites

Docker installed and running.
Dev Containers extension installed (optional).

How to execute

Setup Environment:
- Dev Container (recommended)
- Manually
Open VS Code in the project folder and execute this command in the Command Palette:
Terminal window
> Dev Containers: Reopen in Container
1. Run the setup script to install tools and dependencies.
  Terminal window
  scripts/setup.sh
2. Start Infrastructure: Launch the required containers.
  Terminal window
  docker compose up -d
Run the Example:
- Python
- AWS CLI
Terminal window
python main.py
Upload the README file to the bronze bucket:
Terminal window
aws s3 cp README.md s3://bronze --profile minio

How to debug

main.py:
- Open: Open main.py.
- Breakpoints: Set breakpoints in the code.
- Run: Press F5 to start debugging.
Tests:
- Open: Open a test file (e.g., tests/components/test_s3_boto.py).
- Breakpoints: Set breakpoints in the test code.
- Run: Use the VS Code Testing tab and click the Debug Test icon next to the test you want to debug.

How to test

Individually: Run tests individually from the VS Code Testing tab.
All tests: Run the automated script:
Terminal window
```
scripts/run_tests.sh
```

Validate results

Open:
Terminal window
```
http://localhost:9001
```
Credentials: Use the MINIO_ROOT_USER and MINIO_ROOT_PASSWORD defined in your .env.
Verify: Check the bronze and silver buckets to see the uploaded files.

Run: List the contents of the bronze bucket:
Terminal window
```
aws s3 ls s3://bronze --recursive
```

Enter Shell:
Terminal window
```
scripts/minio_cli.sh
```
Verify:
Terminal window
```
mc ls data/bronze/ --recursive
```

Clean Up

docker compose down -v