Simple ETL

This guide shows you how to work with GCP Cloud SQL and Firebase Cloud Functions locally using Docker Compose and Python. This lab demonstrates how to process a CSV upload via a Cloud Storage trigger and insert records into PostgreSQL.

Prerequisites

• Docker installed and running.
• Dev Containers extension installed (optional).

How to execute

1. Setup Environment:

   Dev Container (recommended)

   Open VS Code in the project folder and execute this command in the Command Palette:

   > Dev Containers: Reopen in Container

   Manually
   1. Start Infrastructure: Launch the PostgreSQL container:

      docker compose up -d postgres

   2. Run the setup script to install tools and dependencies:

      scripts/setup-mve.sh

2. Start Firebase Emulators:

   firebase emulators:start

3. Run the Example:

   python main.py

Validate results

Firebase UI

Open the Firebase Emulator UI and verify the uploaded file and function logs:

http://localhost:4000/storage

http://localhost:4000/functions

SQLTools

Open the SQLTools extension in VS Code and use the preconfigured Postgres connection to run:

SELECT * FROM users;

External Client

Connect using an external client like DBeaver, configure the connection details defined in your .env and run:

SELECT * FROM users;

Clean Up

docker compose down -v

Next steps

• See the AWS equivalent project.
• Dive into the Cloud SQL Lab.
• Dive into the Firebase Cloud Functions Lab.