Airflow

This guide shows you how to work with Apache Airflow locally using Docker. This lab demonstrates a simple ETL pipeline with extract, transform, and load tasks running as a scheduled DAG.

Prerequisites

Docker installed and running.

How to execute

Install dependencies:
Terminal window
```
pip install uv && uv sync
```
Create required directories:
Terminal window
```
mkdir -p dags logs plugins
```
Start Infrastructure: Launch all Airflow services:
Terminal window
```
docker compose up -d
```
Enable DAG:
- Open the Airflow UI in your browser (credentials from .env):
  Terminal window
```
http://localhost:8080
```
- Enable the etl_pipeline DAG by clicking the toggle switch. The DAG runs automatically every 30 seconds.

Validate results

Airflow UI:
- Monitor: Click the DAG name to view the Graph (extract_task → transform_task → load_task).
- Logs: Click a task box, select the task instance, then click Log to inspect ETL output.

Inspect generated files: the files created inside the worker container:

docker exec -it <worker_container_id> ls -la storage/extract/
docker exec -it <worker_container_id> ls -la storage/transform/
docker exec -it <worker_container_id> ls -la storage/load/

Clean Up

docker compose down -v
rm -rf logs/*

Troubleshooting

Issue	Solution
Services not starting	Check logs with `docker compose logs`. Initialization may take 30-60 seconds.
DAG not appearing	Ensure files are in `dags/`, check for Python errors, and refresh the UI.