Databricks
This guide shows you how to simulate a Databricks environment locally using Docker, MinIO as S3-compatible storage, and PostgreSQL as a persistent Hive Metastore. This lab demonstrates how to develop and test Spark and Delta Lake ETLs locally using Databricks Runtime 15.4 LTS (Spark 3.5.0).
Prerequisites
Section titled “Prerequisites”- Docker installed and running.
- Dev Containers extension installed (optional).
How to execute
Section titled “How to execute”-
Setup Environment:
Open VS Code in the project folder and execute this command in the Command Palette:
Terminal window > Dev Containers: Reopen in ContainerLaunch all required services (Spark, MinIO, PostgreSQL):
Terminal window docker compose up -d -
Run the Example:
Terminal window python main.pyExecute the script inside the Spark container:
Terminal window docker compose exec spark python3 main.py
Clean Up
Section titled “Clean Up”docker compose down -vTroubleshooting
Section titled “Troubleshooting”| Issue | Solution |
|---|---|
| Connection Refused | Ensure all services are running: docker compose ps |
| Table already exists | Drop the conflicting table with spark.sql("DROP TABLE IF EXISTS sales.products_silver") or run Clean Up to reset the volumes. |