S3 (RustFS)
This guide shows you how to work with AWS S3 using RustFS as a local emulator. This lab demonstrates a data pipeline using Boto3, PyArrow, and Delta Lake (delta-rs).
Prerequisites
Section titled “Prerequisites”- Docker installed and running.
- Dev Containers extension installed (optional).
How to execute
Section titled “How to execute”-
Setup Environment:
Open VS Code in the project folder and execute this command in the Command Palette:
Terminal window > Dev Containers: Reopen in Container- Start Infrastructure: Launch the required containers.
Terminal window docker compose up -d - Run the setup script to install tools and dependencies.
Terminal window scripts/setup.sh
- Start Infrastructure: Launch the required containers.
-
Run the Example:
Terminal window python main.py
How to debug
Section titled “How to debug”-
main.py:
- Open: Open
main.py. - Breakpoints: Set breakpoints in the code.
- Run: Press
F5to start debugging.
- Open: Open
-
Tests:
- Open: Open a test file (e.g.,
tests/components/test_s3_boto.py). - Breakpoints: Set breakpoints in the test code.
- Run: Use the VS Code Testing tab and click the Debug Test icon next to the test you want to debug.
- Open: Open a test file (e.g.,
How to test
Section titled “How to test”- Individually: Run tests individually from the VS Code Testing tab.
- All tests: Run the automated script:
Terminal window scripts/run_tests.sh
Validate results
Section titled “Validate results”- Open in your browser:
Terminal window http://localhost:9001/rustfs/console/ - Log in: Use the
S3_ACCESS_KEYandS3_SECRET_KEYdefined in your.env. - Verify: Check the
bronzeandsilverbuckets to see the uploaded files.
- Check Buckets: Verify that
bronzeandsilverbuckets were created.Terminal window aws s3 ls --profile rustfsTerminal window mise exec -- aws s3 ls --profile rustfs - Check Content: Verify the files in the silver bucket.
Terminal window aws s3 ls s3://silver --recursive --profile rustfsTerminal window mise exec -- aws s3 ls s3://silver --recursive --profile rustfs
Clean Up
Section titled “Clean Up”docker compose down -vTroubleshooting
Section titled “Troubleshooting”| Issue | Solution |
|---|---|
| RustFS API not ready | Ensure the rustfs service is running and wait a few seconds for the API to initialize. Check logs with docker compose logs rustfs. |
| Port 9000 or 9001 in use | The S3 (MinIO) lab uses the same ports. Stop it before starting this one, or change the mapping in docker-compose.yml. |