A modular data platform for end-to-end analytics, data pipeline orchestration, machine learning model registry in local open-source environments or for a commercial cloud setup.

The project, as an example, used public air-quality data.

The main goal was an educational one.

📸 System Overview

Data Platform Architecture:

Resource Monitoring:

Key Services:
Dagster, Superset, pgAdmin, MinIO, MLflow, JupyterLab, Grafana

🛠️ Service Deployment

Services are deployed via Docker:

Analytics:

• pgAdmin:
  PostgreSQL database web management tool.
  http://localhost:5050
• Superset:
  Data visualization and dashboarding platform.
  http://localhost:8090

Data Science:

• JupyterLab:
  Interactive data analysis notebooks.
  http://localhost:8888

Monitoring Resources:
Documentation.

• Grafana:
  Visualize metrics and build dashboards.
  http://localhost:3000

• Prometheus:
  Metrics collection and time-series storage.
  

• Loki:
  Centralized log aggregation and querying.
  

• Alertmanager:
  Manage and route monitoring alerts.
  

• Promtail:
  Collect and forward container logs.
  

• Caddy:
  Secure reverse proxy and gateway.
  

Orchestration Data Pipelines:

• Dagster:
  Orchestrate and schedule data pipelines.
  http://localhost:5000

Database:

• PostgreSQL:
  Relational database for structured data.
  via client, not web: localhost:5432

Unstructured Data Storage / Datalake:

• MinIO:
  S3-compatible object data storage.
  
  • API: http://localhost:9000
  • Web: http://localhost:9001

Machine Learning Models Registry:

• MLflow:
  Track and manage ML models.
  http://localhost:5005

In-Memory, Key-Value, Database:

• Redis:
  Fast in-memory cache and queue used for Superset.
  

(Optional) Team Collaboration:

• Mattermost:
  Team chat and collaboration platform.
  http://localhost:8065

📦 Requirements

For Services Deployment:

• docker-compose.yml

For Cloud Deployment:

• Terraform: Infrastructure as Code
• Ansible: Automated OS Setup
• Azure: Cloud Service Platform

Tested Operating System: Ubuntu 22.04.5 LTS Linux

⚙️🔨 Installation and Usage

For local use

1. Clone the repository

   git clone https://github.com/Luk-kar/SmogSense.git
   cd SmogSense

2. Configure environment variables

   cp .env.example .env
   # Open .env in your editor and adjust passwords, users or ports etc.

3. Start services with Docker Compose

   docker-compose up --build -d

   ---

4. Verify everything is running

   docker ps
   # You should see:
    # smogsense_postgres, smogsense_pgadmin, smogsense_minio... etc

5. Access the application Open Dagster at:
   http://localhost:5000

   Run example tasks:

   • upload_example_project_data_to_minio
   • restore_example_project_database

6. Troubleshooting
   Check asset logs for service issues

7. Stop all services

   docker compose stop

8. Remove all services from the machine and their resources

   docker compose down -v

☁️ Cloud Deployment

See server_deployment/README.md

Data Pipeline

🌐 Data Sources:

• GIOS API
  Air quality data (stations, measurements, AQI)
• Statistical API (GUS BDL)
  Health and demographic statistics
• Custom JSON Endpoint
  Geospatial province data
• Twitter (X) API
  Social media data

✅ Data Quality:

Assets include automated Dagster checks for validation.

Example validation logic: air_quality/assets/annual_statistics/validations

🏗️ Dagster services setup:

See README.md

🔧 Configuration

• Look at .env.example.
• ⚠️ Avoid storing secrets in .env for production - use dedicated secrets management
• Security tags:
  • #SECRETS: Passwords/API keys requiring secure handling
  • #WARNING: Critical configuration needing attention

✅ Testing

Automated tests for data acquisition features:

tests/run_tests_data_air_quality.sh

or

tests/run_tests_all.sh

💡 Architectural Insight

1. Implementing a mapping table is more efficient than repeatedly dropping/recreating tables to unify schemas from disparate sources.

   Advantages:

   • Schema Preservation: Maintains original source structures
   • Data Integrity: Avoids destructive operations
   • Maintainability: Simplifies schema evolution
   • Performance: Enables incremental updates without redefining foreign keys

   This pattern is particularly valuable in ETL pipelines with independently changing source schemas.

2. Superset dashboard exports may encounter compatibility issues when imported on different machines.

🧩 Further Possible Development

Jenkins Integration

Integrate Jenkins for CI/CD automation.

Workflow:

1. Code Commit: Changes pushed to repo
2. Build Trigger: Jenkins detects changes
3. Test Execution:
   Uses development data environments (MinIO/PostgreSQL) to protect production data
4. Result Reporting: Archived logs and reports
5. Deployment:
   Development: Uses dev data environments
   Production: Moves tested pipelines to production

Benefits:

• Automated testing on commit
• Pipeline-as-code via Jenkinsfile
• Build/test reporting and alerts
• Plugin ecosystem integration

📜 License

MIT License - See LICENSE