Thalassa

Thalassa is a production-style batch data engineering project for Greek maritime traffic analytics.

It uses Bruin to orchestrate a fully scheduled pipeline that ingests public sailing traffic data from the data.gov.gr sailing_traffic API, lands raw records in BigQuery, transforms them into curated analytics tables, and serves the results through a Streamlit dashboard covering operational KPIs, route patterns, and port analysis.

This repository is written to satisfy the spirit of the DE Zoomcamp course project: pick a real dataset, build an end-to-end pipeline, transform the data in a cloud warehouse, and expose it through a dashboard that is easy for reviewers to reproduce.

Table of contents

• Repository layout
• Quick start
• Problem statement and Dataset
• Course project requirements mapping
• Architecture
• Step-by-step reproduction
• Dashboard
• Using AI on Bruin Cloud UI
• Future improvements
• Contributing

Repository layout

• .bruin.yml.example: Bruin connection config template
• .env.example: local environment template
• .streamlit/: copy secrets.toml.example to secrets.toml for service account auth
• dashboard/: Streamlit dashboard
• infra/: Terraform for GCP foundation
• notebooks/: exploration notebooks
• pipeline/: Bruin pipeline
• pyproject.toml: Python project and dependency definitions
• runtime_config.py: shared runtime configuration helpers
• scripts/: operational helpers — dataset sync, snapshot generation, parity checks
• uv.lock: locked dependency versions

Quick start

1. Clone the repository and run uv sync (or uv sync --extra local for dashboard + notebooks).
2. Complete Authenticate to Google Cloud.
3. Run the setup flow in Choose a setup path. If the dataset and infrastructure already exist, use the local-only switch in Change the dataset later instead.
4. Run the initial backfill.
5. Launch the dashboard.

Problem statement and Dataset

Greek coastal traffic data is publicly available, but it is not immediately usable for analytics. Raw API responses are noisy, schema-light, and difficult to compare across time, ports, and routes.

This project answers questions such as:

• Which corridors carry the highest passenger volume?
• Which ports are the busiest over time?
• How do passenger and vehicle volumes change daily, weekly, and monthly?
• Is traffic concentrating in a few corridors?
• Which ports are arrival-heavy or departure-heavy in a given window?

To answer those questions reliably, the project builds a repeatable batch pipeline with explicit quality checks, curated warehouse models, and a dashboard backed by stable fact tables instead of ad hoc raw queries.

Dataset

• Source: data.gov.gr sailing_traffic API
• Domain: Greek maritime passenger and vehicle traffic
• Pipeline interpretation: the source is modeled as reported traffic observations by service date, route code, departure port, and arrival port
• Analytics outcome: curated route- and port-level warehouse tables for dashboarding

Course project requirements mapping

See docs/course_requirements.md for the full mapping of DE Zoomcamp evaluation criteria to this project.

Architecture

See docs/architecture.md for diagrams, pipeline details, tech stack, warehouse layers, partitioning, and data quality checks.

Step-by-step reproduction

Tip

Commands shown use Bash/Linux syntax. PowerShell alternatives are available in collapsibles where the commands differ.

Important

Required tools:

You need these tools available in your shell:

• Python 3.11+
• uv
• bruin
• terraform
• gcloud

Clone the repository

git clone https://github.com/dimzachar/thalassa-analytics.git
cd thalassa-analytics

Install Python dependencies

Install core dependencies only:

uv sync

Install with the Streamlit dashboard and DuckDB local mirror:

uv sync --extra dashboard

Install with Jupyter notebooks support:

uv sync --extra notebooks

Install everything (dashboard + notebooks + DuckDB):

uv sync --extra local

Create or choose a GCP project

You need a GCP project with billing enabled.

At minimum, the project must support:

• BigQuery
• IAM
• Secret Manager
• Service Usage
• optional Artifact Registry

Authenticate to Google Cloud

The simplest local path is Application Default Credentials.

gcloud auth application-default login
gcloud auth application-default set-quota-project YOUR_GCP_PROJECT

Replace YOUR_GCP_PROJECT with your real GCP project ID.

Alternative: use a service account JSON file and export one of these variables:

Linux/macOS:

export GOOGLE_APPLICATION_CREDENTIALS=/absolute/path/to/key.json
# or
export THALASSA_GCP_SERVICE_ACCOUNT_FILE=/absolute/path/to/key.json

Windows (PowerShell):

$env:GOOGLE_APPLICATION_CREDENTIALS = "C:\absolute\path\to\key.json"
# or
$env:THALASSA_GCP_SERVICE_ACCOUNT_FILE = "C:\absolute\path\to\key.json"

For Streamlit specifically, you can also copy .streamlit/secrets.toml.example to .streamlit/secrets.toml and fill in the service account values.

Configure Bruin

If you do not already have a local .bruin.yml, create it from the example first.

Linux/macOS:

cp .bruin.yml.example .bruin.yml

Windows (PowerShell):

Copy-Item .bruin.yml.example .bruin.yml

Then update .bruin.yml so the gcp-default connection points to your project.

Minimal example:

default_environment: default

environments:
  default:
    connections:
      google_cloud_platform:
        - name: gcp-default
          project_id: YOUR_GCP_PROJECT
          location: EU
          use_application_default_credentials: true

Replace YOUR_GCP_PROJECT with your real GCP project ID.

Choose a setup path

Pick one of these and follow only that one.

One-click setup (recommended)

Use this when you want one command to set the dataset, sync Bruin, validate the pipeline, and apply Terraform. Run the command from the repo root after Authenticate to Google Cloud is done. If .bruin.yml does not exist yet, the helper creates it from .bruin.yml.example automatically.

Linux/macOS:

./scripts/set_dataset.sh my_dataset --project-id YOUR_GCP_PROJECT --region europe-west1 --bq-location EU --environment prod

Windows (PowerShell):

.\scripts\set_dataset.ps1 my_dataset -ProjectId YOUR_GCP_PROJECT -Region europe-west1 -BqLocation EU -Environment prod

---

Note

What this does:

• creates .env from .env.example if needed
• sets THALASSA_BQ_DATASET
• sets THALASSA_BQ_PROJECT and THALASSA_BQ_LOCATION when you pass them
• creates .bruin.yml from the example if needed
• updates the gcp-default Bruin connection project and location
• creates infra/terraform.tfvars from the example if needed
• writes project_id, region, bq_location, and environment
• syncs Bruin asset dataset prefixes
• runs bruin validate ./pipeline --fast
• runs terraform init
• selects or creates a dataset-specific Terraform workspace
• runs terraform plan and apply

---

Replace:

• my_dataset with the dataset name you want
• YOUR_GCP_PROJECT with your real GCP project ID
• Change europe-west1, EU, and prod too if your region, BigQuery location, or environment are different.

The dataset name lives in .env. Terraform reads it from THALASSA_BQ_DATASET, so you do not need to repeat it in infra/terraform.tfvars. Project-wide Terraform resource names are derived from that dataset too. For example, my_dataset becomes a service account like my-dataset-pipeline and an Artifact Registry repo like my-dataset.

Add -AutoApprove in PowerShell or --auto-approve in Bash if you want a non-interactive Terraform apply.

Manual setup

If you want to control .env and Terraform inputs yourself, run these commands from the repo root.

Linux/macOS:

cp .env.example .env
cp infra/terraform.tfvars.example infra/terraform.tfvars

Windows (PowerShell):

Copy-Item .env.example .env
Copy-Item infra/terraform.tfvars.example infra/terraform.tfvars

Edit .env and set at least:

• THALASSA_BQ_PROJECT
• THALASSA_BQ_DATASET
• THALASSA_BQ_LOCATION

Then edit infra/terraform.tfvars and set at least:

• project_id
• region
• bq_location
• environment

Then sync Bruin and apply the infrastructure:

uv run --no-project python ./scripts/sync_bruin_dataset.py
bruin validate ./pipeline --fast
terraform -chdir=infra init
terraform -chdir=infra workspace select dataset-my_dataset || terraform -chdir=infra workspace new dataset-my_dataset
terraform -chdir=infra plan
terraform -chdir=infra apply

See more about what Terraform creates.

Bruin local secret placeholders

Those AI secret slots are empty placeholders. You do not need to add OpenRouter, Anthropic, or Gemini keys unless you want LLM-generated intelligence text. Without them, the project still runs and falls back to deterministic reporting.

• The thalassa.auto_panel_snapshot_writer asset expects Bruin secret connection names to exist.
• For local runs without an LLM key, add placeholder generic connections once:

bruin connections add --environment default --name openrouter_api_key --type generic --credentials '{}'
bruin connections add --environment default --name openrouter_model --type generic --credentials '{"value":"google/gemini-2.0-flash-exp:free"}'
bruin connections add --environment default --name openrouter_base_url --type generic --credentials '{"value":"https://openrouter.ai/api/v1"}'

• If your .bruin.yml was created from .bruin.yml.example, these placeholders already exist and you can skip these commands.
• These can be empty; with no API key, snapshot generation falls back to deterministic mode.

Run the initial backfill

For a first run, use --full-refresh. This is especially important if tables already exist and you want BigQuery partitioning and clustering to match the checked-in model definitions.

uv run --no-project python ./scripts/sync_bruin_dataset.py
bruin run --full-refresh ./pipeline/assets/ingestion/raw_sailing_traffic.py --downstream --start-date 2025-01-01 --end-date 2025-01-31 --var 'source_data_lag_days=0'

The pipeline default is source_data_lag_days=1 for scheduled runs. For manual backfills, pass --var 'source_data_lag_days=0' so the requested --start-date and --end-date are used as-is.

To backfill a larger date range, use --var 'request_window_unit="month"' to batch requests by month instead of day. The API data starts from 2017-01-03.

bruin run ./pipeline/assets/ingestion/raw_sailing_traffic.py --downstream --start-date 2017-01-01 --end-date 2023-12-31 --var 'request_window_unit="month"' --var 'source_data_lag_days=0'

Note

The API enforces a 249-day maximum per request window. month (max 31 days) is safe. year will always fail with a 400.

What this does:

• fetches API data for the selected window
• lands raw rows in <THALASSA_BQ_DATASET>.raw_sailing_traffic
• builds staging, intermediate, marts, and report tables
• refreshes <THALASSA_BQ_DATASET>.intelligence_snapshots
• triggers the snapshot writer downstream

Launch the dashboard

uv run streamlit run ./dashboard/app.py

Open the local Streamlit URL shown in the terminal.

If BigQuery returns USER_PROJECT_DENIED / serviceusage.services.use:

• Local Streamlit auth uses your local identity by default (ADC user), unless you explicitly set THALASSA_GCP_SERVICE_ACCOUNT_FILE, GOOGLE_APPLICATION_CREDENTIALS, or .streamlit/secrets.toml.
• Terraform grants roles/serviceusage.serviceUsageConsumer and roles/bigquery.jobUser to the pipeline service account (<dataset>-pipeline), not automatically to every human user in the project.
• The identity used by Streamlit must have permission to use the project for quota/billing (serviceusage.services.use) and BigQuery read access to the configured dataset.

Optional: visualize the pipeline DAG

Install Bruin Visualizer (source) for an interactive DAG view with impact analysis and run history:

uv tool install bruin-visualizer

# Parse the pipeline (generates pipeline_graph.json)
bruin-viz parse ./pipeline

# Open the visualizer at http://localhost:8001
bruin-viz serve

To also record run history before serving:

bruin-viz run ./pipeline --start-date 2025-01-01 --end-date 2025-01-31 --workers 1
bruin-viz parse ./pipeline
bruin-viz serve

Change the dataset later

Once the project is already set up, use the same script to switch datasets.

For a local-only switch without Terraform:

Linux/macOS:

./scripts/set_dataset.sh my_dataset --skip-terraform

Windows (PowerShell):

.\scripts\set_dataset.ps1 my_dataset -SkipTerraform

Replace my_dataset with the dataset name you want.

If the new dataset also needs Terraform changes applied, rerun the command from One-click setup (recommended) without the skip flag.

Validate manually

uv run --no-project python ./scripts/sync_bruin_dataset.py
bruin validate ./pipeline --fast
bruin validate ./pipeline

Optional: regenerate intelligence snapshots directly

These commands are useful when you want to refresh only the narrative layer.

bruin run ./pipeline/assets/reports/intelligence_snapshots.sql --downstream

or

uv run python ./scripts/generate_auto_panel_snapshot.py

Verify that the run worked

At this point you should have:

• BigQuery tables under your configured THALASSA_BQ_DATASET
• a dashboard that loads KPI cards and charts successfully
• visible categorical and temporal tiles for course review

A quick warehouse sanity query is:

SELECT
  MAX(service_date) AS latest_service_date,
  SUM(total_passengers) AS passengers,
  SUM(total_vehicles) AS vehicles
FROM `YOUR_GCP_PROJECT.YOUR_THALASSA_BQ_DATASET.row_counts_daily`;

Replace YOUR_GCP_PROJECT and YOUR_THALASSA_BQ_DATASET with your real values.

For a full set of ready-to-run inspection queries (row counts, date ranges, top routes, top ports, and more), see docs/thalassa_queries.sql.

Delete the GCP resources

See more about how cleanup works.

terraform -chdir=infra workspace select dataset-YOUR_THALASSA_BQ_DATASET
terraform -chdir=infra destroy

If you want Terraform to delete the dataset contents too:

1. set dataset_delete_contents_on_destroy = true in infra/terraform.tfvars
2. run terraform -chdir=infra workspace select dataset-YOUR_THALASSA_BQ_DATASET
3. run terraform -chdir=infra destroy

Dashboard

The dashboard reads curated BigQuery tables, not raw API payloads. The default view window is the last 90 days, giving an at-a-glance picture of recent traffic.

Dashboard query caching is data-aware: cached reads are automatically invalidated when key warehouse tables (row_counts_daily, fct_route_traffic_daily, fct_port_activity_daily, intelligence_snapshots) are updated.

The main page shows:

• KPI cards for total passengers, total vehicles, active corridors, traffic concentration, and the leading port
• Traffic Pulse: a daily trend chart for passengers, vehicles, and sailings over the view window
• Top Corridors: a ranked view of the busiest departure-arrival pairs by passenger volume
• Port Balance: a ranked view of the busiest ports by total passenger flow
• (Optional) A cached intelligence panel backed by <THALASSA_BQ_DATASET>.intelligence_snapshots — see docs/agent_panel.md for how the snapshot pipeline, LLM gating, and fallback logic work

The Analytics page lets users filter by date range, port, and corridor to explore:

• weekday traffic profile
• corridor concentration over time
• corridor efficiency metrics
• port net flow (arrivals vs departures)

This means the course dashboard requirement is covered by at least:

• one temporal tile: Traffic Pulse
• one categorical tile: Top Corridors or Port Balance

Using AI on Bruin Cloud UI

If you're already running this pipeline, you can connect it to Bruin Cloud and query your data conversationally through a built-in AI agent.

Connect the project:

Go to Team Settings → Projects and add this repo. Once linked, enable the pipeline to run and backfill from the Cloud UI.

Create an AI agent:

Go to Agents → Create New Agent, select the repo and configure any credentials you need (e.g. Slack). You will also need to configure connections and then start using it.

What you can do:

• Chat with the agent inside the UI to ask questions about the data
• Review the SQL it generates before it runs
• Connect to Slack and query from there too
• Ask the agent to generate a PDF report directly in chat

Pipeline monitoring:

The Cloud UI also doubles as a pipeline dashboard — you can track runs, failures, and lineage without leaving the browser.

Scheduled lag setting on Bruin Cloud:

• The pipeline default is source_data_lag_days=1 because the source publishes data with a one-day lag, so scheduled runs target the latest available complete date.
• For manual backfills where you want literal dates, pass --var 'source_data_lag_days=0'.
• Re-running the same interval with strategy: append can append duplicate rows in raw_sailing_traffic (downstream models deduplicate by record_hash).

Enable LLM overlay in scheduled pipeline snapshots:

• Create three Generic Secret connections in Bruin Cloud:
• openrouter_api_key → your OpenRouter API key
• openrouter_model → your preferred OpenRouter model id (optional)
• openrouter_base_url → https://openrouter.ai/api/v1
• The thalassa.auto_panel_snapshot_writer asset injects these as OPENROUTER_API_KEY, OPENROUTER_MODEL, and OPENROUTER_BASE_URL.
• Verify from run logs: mode=llm_overlay indicates the LLM path was used.

Future improvements

• Add CI checks for bruin validate, dependency sync, and Terraform planning
• Add automated tests
• Add monitoring

Contributing

Contributions are welcome, especially around data quality, dashboard UX, testing, and deployment hardening.

Before opening a PR:

1. Keep the change focused on one concern
2. Run uv sync (or uv sync --extra local) if dependencies changed
3. Run uv run --no-project python ./scripts/sync_bruin_dataset.py
4. Run bruin validate ./pipeline
5. If you changed SQL models or ingestion logic, run the relevant bruin run command for a representative date window
6. If you changed the dashboard, include a screenshot or short note describing the UI impact
7. In the PR description, include the purpose of the change, touched paths, and validation commands you ran

For larger changes, opening an issue first is the best way to align on scope before implementation.