Databricks

This guide shows you how to simulate a Databricks environment locally using Docker, MinIO as S3-compatible storage, and PostgreSQL as a persistent Hive Metastore. This lab demonstrates how to develop and test Spark and Delta Lake ETLs locally using Databricks Runtime 15.4 LTS (Spark 3.5.0).

Prerequisites

Docker installed and running.
Dev Containers extension installed (optional).

How to execute

Setup Environment:
- Dev Container (recommended)
- Manually
Open VS Code in the project folder and execute this command in the Command Palette:
Terminal window
> Dev Containers: Reopen in Container
1. Start Infrastructure: Launch all required services (Spark, MinIO, PostgreSQL).
  Terminal window
  docker compose up -d
2. Run the setup script inside the Spark container to install tools and dependencies.
  Terminal window
  docker compose exec spark bash scripts/setup.sh
Run the Example:
- Dev Container (recommended)
- Manually
Terminal window
python main.py
Execute the script inside the Spark container:
Terminal window
docker compose exec spark .venv/bin/python main.py

How to debug

Python Script (main.py):
- Open main.py and set breakpoints.
- Run the Python: Main configuration from the Run and Debug tab.
Interactive Notebook:
- Open src/notebooks/analysis.ipynb and debug cells with the Jupyter debugger.

How to test

All tests: Run the automated script: scripts/run_tests.sh
Individually: Use the VS Code Testing tab to run or debug specific test cases.

While a job is running, Open the Spark UI:

http://localhost:4040

Browse the Delta files written to the bucket. Log in with MINIO_ROOT_USER / MINIO_ROOT_PASSWORD:

http://localhost:9001

Clean Up

docker compose down -v

Troubleshooting

Issue	Solution
Connection Refused	Ensure all services are running: `docker compose ps`
Table already exists	Drop the conflicting table with `spark.sql("DROP TABLE IF EXISTS sales.products_silver")` or run Clean Up to reset the volumes.