Popular repositories Loading

1. data-lakehouse-pipeline data-lakehouse-pipeline Public

   End-to-end batch lakehouse: NYC Taxi → Delta Lake medallion (bronze/silver/gold) with Airflow + PySpark, verified end-to-end in CI.

   Python

2. realtime-streaming-system realtime-streaming-system Public

   Real-time pipeline: Kafka → Spark Structured Streaming → ClickHouse → Grafana with windowed aggregation and anomaly detection.

   Python

3. data-quality-observability data-quality-observability Public

   Data quality + observability: Great Expectations validation, OpenLineage lineage, Z-score/IQR anomaly detection, Slack SLA alerting.

   Python

4. data-platform-iac data-platform-iac Public

   Infrastructure-as-code for a data platform: Terraform AWS modules (MSK/RDS/Redshift/S3) + a one-command local Docker Compose stack.

   HCL

5. self-service-data-platform self-service-data-platform Public

   Self-service data platform: Jinja pipeline scaffolding, a Delta data catalog, and reusable connectors via a Click CLI + FastAPI.

   Python

6. data-cost-optimizer data-cost-optimizer Public

   SQL cost optimizer: sqlglot anti-pattern analysis, BigQuery/Snowflake/Spark cost estimates, partition/Z-order advice, engine benchmarks.

   Python