🧠 Developer Knowledge Engine

An automated ETL pipeline and AI-driven knowledge graph builder that turns unstructured documentation into queryable relational data.

📖 Overview

Developer Knowledge Engine solves the problem of fragmented documentation by actively scraping developer repositories and constructing an intelligent, interactive Knowledge Graph. By leveraging OpenAI Large Language Models (LLMs) and strict Pydantic schemas, the system extracts critical architectural entities and relationships, transforming raw text into an O(1) queryable structure.

✨ Key Features

• 🔄 Automated ETL Pipeline: Extracts unstructured markdown files directly from remote GitHub repositories and standardizes them locally.
• 🤖 AI Entity Extraction: Utilizes LLMs to read through documentation and intelligently extract technical terms, design patterns, and relational data.
• 🛡️ Strict Data Validation: Ensures zero hallucinations in graph structure by forcing AI outputs through predefined Pydantic validation schemas.
• 🕸️ Relational Knowledge Graph: Constructs an interconnected, interactive graph utilizing NetworkX and stores the persistent data locally via SQLite.
• 📊 Visual Interface: Allows developers to seamlessly navigate complex relationships and search through the parsed knowledge base.

🏗️ Architecture Workflow

[ GitHub Repository ] --> (Scraping via API) --> [ Raw Markdown ]
                                                        |
                                                        V
[ SQLite Storage ] <-- (Pydantic Validation) <-- [ OpenAI LLM Parsing ]
         |
         V
[ NetworkX Graph Builder ] --> [ Interactive Streamlit Frontend ]

🚀 Getting Started

Prerequisites

• Python 3.9+
• OpenAI API Key
• GitHub Personal Access Token

Installation

1. Clone the repository:

   git clone https://github.com/Justinvcj/Developer-Knowledge-Engine.git

2. Install dependencies:

   pip install -r requirements.txt

3. Set your environment variables in .env:

   OPENAI_API_KEY=your_openai_key
   GITHUB_TOKEN=your_github_token

4. Run the ETL Pipeline:

   python pipeline.py

5. Launch the Visualizer:

   streamlit run app.py

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.