🌊 AetherLake — Open-Source Data Lakehouse on Kubernetes

Storage, catalog, query, streaming, BI and identity — one helm install.
Quick Start · Components · Streaming · dbt · SSO · Configuration

✨ What is AetherLake?

A batteries-included, Kubernetes-native data lakehouse that glues best-in-class open-source components into one platform: S3 storage, an Iceberg REST catalog, federated SQL, stream processing, orchestration, BI and centralized identity — managed from a single web Control Panel.

🏗️ Modular — every component behind a single values.yaml toggle
🔐 Secure by default — Keycloak SSO for every UI, random per-install secrets
🌊 Streaming — Kafka (Strimzi) + Flink SQL jobs, queryable from Trino
🔄 dbt Lakehouse — Medallion modeling (Bronze → Silver → Gold) with visual DAG Lineage
🎛️ Unified control — Next.js Control Panel (EN/TR) for status, SQL, catalogs, Kafka, Flink, and dbt

🏛️ Architecture

                        *.aetherlake.local (nginx ingress)
                        │  oauth2-proxy gate (Keycloak SSO)
                        │  for UIs without native OIDC
   ┌────────────────────┼───────────────────────────────────────┐
   │  aetherlake ns     ▼                                       │
   │  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌───────────────┐ │
   │  │ MinIO    │ │ Trino    │ │ Polaris  │ │ Control Panel │ │
   │  │ storage  │◄ SQL +    │ │ Iceberg  │ │ Next.js (SSO) │ │ │
   │  │          │ │ kafka cat│ │ REST cat │ └───────────────┘ │ │
   │  └──────────┘ └────▲─────┘ └──────────                   │ │
   │                    │ queries                              │ │
   │  ┌────────── ┌────┴─────┐ ┌──────────┐ ┌───────────────┐ │ │
   │  │ Kafka    │◄ Flink    │ │ Airflow  │ │ Superset (SSO)│ │ │
   │  │ Strimzi  │ │ SQL jobs │ │ (SSO)    │ │ Milvus/Attu   │ │ │
   │  │ +external│ └────────── └──────────┘ └───────────────┘ │ │
   │  │ SCRAM    │                                             │ │
   │  └──────────┘ ┌─────────────────────────────────────────┐ │ │
   │               │ Keycloak — realm aetherlake, OIDC SSO   │ │ │
   │               └─────────────────────────────────────────┘ │ │
   └────────────────────────────────────────────────────────────┘

Per-component deep dives (settings, diagrams, operations) live in docs/guide/components/.

📦 Components

Component	Role	Version
Keycloak	Identity & SSO (OIDC)	26.3.3
MinIO	S3-compatible object storage	Operator tenant
Trino	Federated SQL (Iceberg + Kafka catalogs)	480
Apache Polaris	Iceberg REST catalog	Postgres metastore
Apache Kafka	Streaming (Strimzi, KRaft)	4.3.0
Apache Flink	Stream processing (SQL jobs)	2.1 / Operator 1.15
dbt	Medallion modeling & Lineage	1.8 (dbt-trino)
Apache Airflow	Orchestration	2.10.5
Apache Superset	BI & dashboards	3.1.2
Apache Spark	Batch processing	Operator 1.1.27
Milvus	Vector search	chart 5.0.14
PostgreSQL	Metadata stores	16
Control Panel	Platform UI (Next.js)	EN/TR

🚀 Quick Start

Prerequisites: Kubernetes (Docker Desktop / minikube / kind), Helm ≥ 3.12, kubectl, Docker (the installer builds the Flink SQL runner image), an NGINX ingress controller (the installer installs one if missing).

git clone https://github.com/mrtozkl/AetherLake.git && cd AetherLake
./install.sh

Add local DNS entries:

127.0.0.1  minio.aetherlake.local trino.aetherlake.local polaris.aetherlake.local
127.0.0.1  keycloak.aetherlake.local airflow.aetherlake.local superset.aetherlake.local
127.0.0.1  milvus.aetherlake.local oauth2.aetherlake.local

Service	URL	Auth
Control Panel	`http://localhost:3000`	dev login `admin`/`admin` (local dev only)
Trino UI	`http://trino.aetherlake.local`	Keycloak SSO
Milvus (Attu)	`http://milvus.aetherlake.local`	Keycloak SSO
MinIO Console	`http://minio.aetherlake.local`	Keycloak OIDC
Airflow	`http://airflow.aetherlake.local`	Keycloak OIDC
Superset	`http://superset.aetherlake.local`	Keycloak OIDC
Keycloak	`http://keycloak.aetherlake.local`	admin (secret)

All credentials are randomly generated into aetherlake-credentials:

kubectl get secret aetherlake-credentials -n aetherlake \
  -o jsonpath='{.data.realm-admin-password}' | base64 -d   # SSO admin (change on first login)

Every host is also served over TLS with a self-signed CA (cert-manager); plain HTTP stays on because the SSO issuer URLs are http://.

🎛️ Control Panel

A unified web console built with Next.js 16 (Turbopack, TypeScript, Tailwind CSS) providing centralized platform visibility and operations:

Overview — pod health, restarts, memory/CPU usage, and one-click service restarts
dbt Workspace & Lineage — interactive DAG graph (Bronze → Silver → Gold), model inspector, Monaco SQL viewer, and run triggers
Kafka — KRaft cluster status, broker readiness, topics (partitions, replicas, configs, and conditions)
Flink SQL — interactive streaming workspace: topic explorer, Monaco SQL editor, job submission, and live status
SQL IDE — federated Trino queries with schema tree explorer across Iceberg and Kafka catalogs
Iceberg Tables & Catalogs — explore Polaris namespaces, table schemas, snapshots, and partition metadata
Observability — live container logs, Kubernetes events, and detailed pod metrics
i18n & RBAC — bilingual (English/Turkish) with role-based action gating (data-admin, data-scientist, data-engineer)

cd control-panel && npm install && npm run dev   # → http://localhost:3000

🔄 dbt Lakehouse Transformations

Transform raw data using the Medallion Architecture (pipelines/dbt/):

Bronze (Raw): Clickstream (user_events) and sensor data (telemetry_stream) landed via Kafka and Flink.
Silver (Curated): Cleansed and partitioned Parquet Iceberg tables (stg_user_events, stg_users).
Gold (Marts): Aggregated metrics and dimensional marts (fct_daily_user_metrics, fct_event_summary) consumed by Superset and Trino.

cd pipelines/dbt
dbt run --profiles-dir .
dbt test --profiles-dir .

🌊 Streaming: Kafka + Flink

Enable with kafka.enabled / flink.enabled (both default true).

Kafka (KRaft Mode): Provisioned by Strimzi 1.1.0 with a pre-configured events topic; Flink SQL jobs produce/consume topics via the built-in Kafka connector (pipelines/flink/examples/).
Flink SQL Runner: Each submission creates an isolated application-mode FlinkDeployment mini-cluster using aetherlake/flink-sql-runner:flink-2.1 built by install.sh.
Kafka → Iceberg Lakehouse Bridge: Continuous streaming ETL from Kafka topics directly into Apache Iceberg tables via Polaris REST catalog and MinIO S3FileIO (pipelines/flink/examples/kafka-to-iceberg.sql). Platform credentials (POLARIS_CREDENTIAL, MINIO_ACCESS_KEY, MINIO_SECRET_KEY) are dynamically injected and resolved via ${ENV:...} placeholders.
Kafka in Trino: Topics queryable directly as SQL tables via SELECT * FROM kafka.aetherlake.events (configured in trino.kafka.tableDescriptions).

Producing from outside the cluster

The external listener (TLS + SCRAM-SHA-512, nodeport) accepts authenticated clients; credentials live in the external-producer KafkaUser secret:

NODEPORT=$(kubectl get svc aetherlake-kafka-external-bootstrap -n aetherlake \
  -o jsonpath='{.spec.ports[0].nodePort}')
kubectl get secret aetherlake-cluster-ca-cert -n aetherlake \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > cluster-ca.crt      # TLS truststore
kubectl get secret external-producer -n aetherlake \
  -o jsonpath='{.data.sasl\.jaas\.config}' | base64 -d            # sasl.jaas.config value

Client properties (localhost:$NODEPORT on Docker Desktop):

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=<value from the secret>
ssl.truststore.location=<truststore built from cluster-ca.crt>
ssl.truststore.password=<truststore password>

Full recipe (truststore build, console producer/consumer examples): docs/guide/components/kafka.md.

🔐 Single Sign-On

Keycloak realm aetherlake with OIDC clients per service. Apps with native OIDC (Superset, Airflow, MinIO) log in directly; UIs without one (Trino, Milvus/Attu) are gated by oauth2-proxy through nginx external auth — one Keycloak login, shared *.aetherlake.local session.

Client	Used by
`aetherlake-client`	Control Panel (NextAuth)
`oauth2-proxy`	SSO gate for Trino UI & Milvus Attu
`superset` / `airflow` / `minio`	native OIDC apps
`polaris`	catalog token issuance

Realm roles: data-admin, data-scientist, data-engineer.

⚙️ Configuration

Component toggles in helm-charts/core-data-stack/values.yaml:

kafka:
  enabled: true
  external:            # authenticated (SCRAM) access from outside the cluster
    enabled: true
flink:
  enabled: true
trino:
  enabled: true
  server:
    workers: 2
airflow:
  enabled: false       # disable anything you don't need

Secrets: everything reads from aetherlake-credentials / open-lake-credentials (random per install; re-runs backfill missing keys). Storage: MinIO tenant (minio.*: servers, volumes, initBuckets).

🤖 MCP Server

AI assistants (Claude, Cursor, …) can operate the platform via mcp-server/: platform status, service logs/restarts, Trino queries, Polaris catalogs, Airflow DAGs. Build with npm install && npm run build, then point your MCP client at mcp-server/dist/index.js (see mcp-server/README.md for the config block).

📁 Project Structure

├── control-panel/        # Next.js 16 UI (overview, kafka, flink, telemetry, …)
├── helm-charts/
│   ├── core-data-stack/  # Data infra chart (+ values-aws.yaml, values-azure.yaml)
│   └── security-stack/   # Keycloak + realm/OIDC provisioning
├── terraform/
│   ├── aws/              # AWS EKS, S3 Lakehouse, IAM IRSA, RDS PostgreSQL
│   └── azure/            # Azure AKS, ADLS Gen2, Workload Identity, Flexible PG
├── mcp-server/           # MCP tools for AI assistants
├── pipelines/            # Airflow DAGs, Spark, Flink SQL-runner + examples, dbt
├── docs/                 # Multi-cloud deployment guides & telemetry reference
├── aetherlake-ingress.yaml
└── install.sh

☁️ Cloud Deployments (AWS & Azure)

AetherLake is cloud-native and ready for production deployment on major cloud providers:

Amazon Web Services (AWS): Amazon EKS + Amazon S3 + AWS IAM IRSA (IAM Roles for Service Accounts) + Amazon RDS PostgreSQL.
- Terraform Module: terraform/aws/
- Helm Profile: helm-charts/core-data-stack/values-aws.yaml
- Deployment Guide: docs/guide/cloud-aws.md
Microsoft Azure: Azure Kubernetes Service (AKS) + Azure Data Lake Storage Gen2 (ADLS Gen2) + Azure Workload Identity + Azure PostgreSQL Flexible Server.
- Terraform Module: terraform/azure/
- Helm Profile: helm-charts/core-data-stack/values-azure.yaml
- Deployment Guide: docs/guide/cloud-azure.md

📡 Anonymous Telemetry

AetherLake includes an anonymous heartbeat to track installation counts, detected cloud environments, and component health. It never collects PII, table schemas, or dataset contents. You can opt-out at any time via telemetry.enabled=false in Helm or TELEMETRY_ENABLED=false. See docs/guide/telemetry.md for full disclosure.

🗺️ Roadmap

AWS EKS + S3 + IAM IRSA Terraform & Helm Stack
Azure AKS + ADLS Gen2 + Workload Identity Terraform & Helm Stack
Anonymous Installation & Health Telemetry
Grafana + Prometheus stack
Apache Ranger policies · lineage UI · multi-cluster federation
GitOps (ArgoCD) · automated backups · chart on an artifact registry

🚨 Security & Production Readiness

Credentials are random per install and the Control Panel refuses to start in production without NEXTAUTH_SECRET / KEYCLOAK_CLIENT_SECRET. Before exposing a cluster: review placeholder values in chart values.yaml, set the Control Panel env vars, override the MCP server's AIRFLOW_AUTH, and use real TLS certificates on the ingress.

🤝 Contributing

Fork → branch → commit → PR. Chart changes must pass helm lint; Control Panel changes must pass npm run build; update docs with your change.

📄 License

BUSL-1.1 — free to use, modify and self-host internally; commercial hosted/managed offerings require a license (see LICENSE). Converts to Apache-2.0 four years after each release. Third-party components keep their own licenses — see THIRD_PARTY_LICENSES.md.

Name		Name	Last commit message	Last commit date
.agent/rules		.agent/rules
.claude		.claude
.github/workflows		.github/workflows
assets		assets
control-panel		control-panel
docs		docs
helm-charts		helm-charts
mcp-server		mcp-server
pipelines		pipelines
scripts		scripts
terraform		terraform
.gitignore		.gitignore
AGENTS.md		AGENTS.md
GEMINI.md		GEMINI.md
LICENSE		LICENSE
README.md		README.md
THIRD_PARTY_LICENSES.md		THIRD_PARTY_LICENSES.md
aetherlake-ingress.yaml		aetherlake-ingress.yaml
aetherlake-tls.yaml		aetherlake-tls.yaml
airflow-values.yaml		airflow-values.yaml
install.sh		install.sh

Navigation Menu

Repository files navigation

🌊 AetherLake — Open-Source Data Lakehouse on Kubernetes

✨ What is AetherLake?

🏛️ Architecture

📦 Components

🚀 Quick Start

🎛️ Control Panel

🔄 dbt Lakehouse Transformations

🌊 Streaming: Kafka + Flink

Producing from outside the cluster

🔐 Single Sign-On

⚙️ Configuration

🤖 MCP Server

📁 Project Structure

☁️ Cloud Deployments (AWS & Azure)

📡 Anonymous Telemetry

🗺️ Roadmap

🚨 Security & Production Readiness

🤝 Contributing

📄 License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

🌊 AetherLake — Open-Source Data Lakehouse on Kubernetes

✨ What is AetherLake?

🏛️ Architecture

📦 Components

🚀 Quick Start

🎛️ Control Panel

🔄 dbt Lakehouse Transformations

🌊 Streaming: Kafka + Flink

Producing from outside the cluster

🔐 Single Sign-On

⚙️ Configuration

🤖 MCP Server

📁 Project Structure

☁️ Cloud Deployments (AWS & Azure)

📡 Anonymous Telemetry

🗺️ Roadmap

🚨 Security & Production Readiness

🤝 Contributing

📄 License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages