Intelligence - Ground 0

[illuminati1618]

---

1. Forming and Operating as a Team

What Is Working

The strongest part of the team process so far is that the project is beginning to have a shared technical direction instead of everyone exploring unrelated ideas.

The OCS AI problem naturally creates different areas that can be owned by different people:

Workstream	Example Responsibilities
Cloud / API	EC2, NGINX, FastAPI, authentication, public endpoint
Networking	NetBird deployment, peer connectivity, P2P vs relay testing
Mini Workers	Worker image, Docker, session/context logic, health reporting
GPU Runtime	Ubuntu, NVIDIA drivers, CUDA compatibility, llama.cpp, model benchmarks
Broker / Scheduler	Worker registry, queues, session assignment, GPU selection, failover
Observability	Health checks, logs, queue depth, latency, GPU utilization
Documentation / Testing	Issues, setup guides, architecture diagrams, experiment results

This structure helps team formation because people can contribute through different strengths while still working on one architecture.

What Still Needs Improvement

The project needs to become more explicit about execution. A technically strong plan is not enough if tasks do not have owners, acceptance criteria, and evidence.

For the next sprint, I want each task to answer:

Question	Example
Who owns this?	One named student / pair
What is the exact output?	A working Mini heartbeat endpoint
How do we know it works?	Broker shows Mini online/offline within a defined timeout
What evidence is required?	Commit, screenshot/log, short benchmark/result table
What depends on it?	Session scheduler cannot work until worker health exists
What is the fallback?	Manual assignment before automatic scheduling is implemented

Team Execution Loop

flowchart LR
    A[Stand-up] --> B[Choose Small Task]
    B --> C[Assign Owner]
    C --> D[Build / Research]
    D --> E[Test]
    E --> F[Post Evidence]
    F --> G[Peer Review]
    G --> H[Integrate]
    H --> I[Update Board / Issue]
    I --> A

Loading

---

2. Beginning the Project — OCS Distributed AI Infrastructure

Project Problem

The project is trying to answer a bigger question than “Can a GTX 1070 run an LLM?”

The actual project is closer to:

Can a heterogeneous pool of low-cost compute nodes and older GPUs be combined into a secure, reliable, low-latency AI coding platform for distributed OCS students?

This turns the hardware into a meaningful research platform rather than a collection of machines.

Hardware Direction

Environment	Mini Layer	GPU Layer	Purpose
Production	~10–15 Mac Minis	Rig 1: 8× GTX 1070	Stable student-facing environment
Development / Test	~10–15 Mac Minis	Rig 2: 8× GTX 1070	Experimentation, scheduler changes, model/runtime testing

The two pools give us a path for testing changes without treating the production environment as the experiment.

---

3. Architecture I Can Now Explain

Control Plane vs Compute Plane

One of the most important things I learned is the difference between the control plane and the compute plane.

Control Plane

The control plane decides what should happen and where it should happen.

OCS CONTROL PLANE
├── Users
├── Roles / Authentication
├── Projects
├── Persistent Sessions
├── API
├── RDS
├── Redis
└── OCS Broker / Scheduler

Compute Plane

The compute plane actually performs the work.

OCS COMPUTE PLANE
├── Mac Mini Pool
│   ├── Session workers
│   ├── Context workers
│   ├── Agent workers
│   └── Request preparation
│
└── GPU Pool
    ├── Rig 1
    └── Rig 2
        ├── Model loading
        ├── Inference
        ├── VRAM management
        └── Model cache

Durable, Fast, and Working State

Another major architecture improvement was separating data by how long it needs to live.

State Type	Technology / Location	Examples	Why
Durable State	RDS / persistent DB	users, sessions, history, project references, request history	Must survive restarts and machine failure
Fast Live State	Redis	active Mini registry, active GPU registry, job queue, assignments, availability, locks	Needs very fast reads/writes and changes constantly
Working State	Mac Mini	current context, recent files, temporary results, local cache	Useful while a session is active but disposable/rebuildable

This separation makes failover possible. A Mini should not become the only machine that knows a student's permanent history.

---

4. End-to-End Request Flow

A student request should eventually follow this path:

sequenceDiagram
    participant S as Student
    participant W as OCS Web/API
    participant B as Broker
    participant R as Redis/RDS
    participant M as Mac Mini Worker
    participant G as GPU / llama-server

    S->>W: Submit question + project/session info
    W->>B: Create inference job
    B->>R: Check session + live worker state
    R-->>B: Session info + available workers
    B->>M: Assign session/job
    M->>M: Prepare context/files
    M->>B: Request GPU capacity
    B->>R: Check GPU availability/model state
    R-->>B: Candidate GPU(s)
    B->>G: Assign inference job
    M->>G: OpenAI-compatible inference request
    G-->>M: Stream model response
    M-->>W: Stream prepared result
    W-->>S: Display response
    W->>R: Persist important session/history state

Loading

This is much clearer to me now than the original idea of simply exposing the GPU rig directly to every student.

---

5. Security / Networking Research

NetBird's Role

NetBird is not the student-facing application. Its role is to provide a private network between infrastructure components.

The desired path is:

Student
   |
 HTTPS
   v
Public OCS Endpoint / AWS
   |
 NetBird Private Overlay
   v
Mac Minis + GPU Rigs

For infrastructure peers, the ideal NetBird path is direct peer-to-peer WireGuard when NAT traversal succeeds:

flowchart LR
    A[AWS / Worker] <-->|Encrypted P2P WireGuard| B[On-Premise Peer]

Loading

A relay is a valid fallback, but it introduces another hop and becomes important to measure because it may affect latency or throughput:

flowchart LR
    A[AWS / Worker] <-->|Encrypted| B[NetBird Relay]
    B <-->|Encrypted| C[On-Premise Peer]

Loading

Networking Research Question

Can the compute resources be securely and reliably exposed to distributed agents without requiring a traditional VPN-gateway architecture, while maintaining acceptable latency and throughput?

Evidence I Want to Collect

Test	Measurement
Direct peer connectivity	Did P2P establish or did the connection relay?
Ping / RTT	Baseline network latency
API latency	Request time without model generation
Throughput	Transfer speed for project context/files
Reliability	Connection behavior across restarts / NAT changes
Failure behavior	What happens when a peer disappears?

---

6. GPU Runtime Research

The GTX 1070 hardware is a project constraint, not just a resource.

The current plan is to treat the GPU runtime as an experiment rather than assume that a modern datacenter stack will support older Pascal hardware cleanly.

Planned Baseline Stack

flowchart TD
    A[Ubuntu Linux] --> B[NVIDIA Driver]
    B --> C[CUDA-Compatible Environment]
    C --> D[llama.cpp]
    D --> E[llama-server]
    E --> F[GGUF Model]
    F --> G[GTX 1070]

Loading

The project plan currently favors llama.cpp + llama-server for the baseline and treats newer GPU-serving stacks such as vLLM as something that should not be assumed to support the GTX 1070 environment.

GPU Research Question

Which software stack gives useful and reliable inference performance on Pascal-era GTX 1070 GPUs in the current environment?

Benchmark Evidence I Want

Category	Example Evidence
Model	model name, parameter count, quantization
Runtime	llama.cpp build/commit, CUDA settings
GPU allocation	1 GPU, multiple independent GPUs, grouped GPUs
Load time	seconds to load model
Prompt speed	prompt tokens/sec
Generation speed	output tokens/sec
VRAM usage	per GPU
Latency	time to first token
Stability	crashes, OOM, driver/runtime failures

---

7. Broker / Scheduler Research

The broker is the part of the project that turns a collection of computers into a system.

Two Scheduling Decisions

The project has two different routing problems:

flowchart TD
    A[New Student Request] --> B{Which Mini?}
    B --> C[Mini 04]
    C --> D[Prepare Session + Context]
    D --> E{Which GPU?}
    E --> F[GPU 2]
    F --> G[Inference]

Loading

Mini Scheduling

The broker needs to know:

• Which Minis are alive?
• Which Minis are overloaded?
• Which Mini owns a session?
• Should a returning session go back to the same Mini?
• What happens if a Mini disappears?
• Can session working state be recreated elsewhere?

GPU Scheduling

The GPU layer needs to know:

• Which GPUs are healthy?
• Which GPUs are busy?
• Which model is already loaded where?
• How much VRAM is available?
• Should a request wait or use a different model/GPU?
• When is grouping multiple GPUs worth the coordination cost?
• What happens if an entire rig disappears?

Initial Scheduler Strategy

The first scheduler should be intentionally simple. I do not want to begin with a “smart” scheduler that is impossible to validate.

MVP rule:

1. Filter to healthy workers.
2. Filter to workers that can run the requested model/job.
3. Prefer an already-loaded model if possible.
4. Choose the least-busy candidate.
5. Queue the request if no candidate is available.
6. Log the decision so it can be evaluated later.

Then we can compare smarter policies against that baseline.

---

8. Phased Implementation Plan

One of the most important project decisions is not trying to deploy all 30 Minis and 16 GPUs at once.

Phase 0 — Ground Zero / Architecture

Goal: Understand the problem and define the system.

• Identify available hardware roles
• Separate cloud control plane from local compute plane
• Identify NetBird as private networking layer
• Identify Mac Minis as session/context/agent workers
• Identify GPU rigs as inference workers
• Define RDS vs Redis vs working state
• Identify llama.cpp / llama-server baseline direction
• Identify need for Mini and GPU scheduling
• Document research questions and architecture
• Convert architecture into small testable service contracts

Phase 1 — Smallest End-to-End Proof

Goal: One request passes through the entire planned path.

Client
  ↓
FastAPI
  ↓
NetBird
  ↓
1 Mini
  ↓
1 GPU
  ↓
llama-server
  ↓
response

Acceptance criteria:

• Public/private API can accept a simple request
• AWS/control node can reach Mini over NetBird
• Mini can reach GPU inference endpoint
• GPU returns a model response
• Response successfully returns to client
• Logs show every hop
• Basic latency is recorded

Phase 2 — Worker Registration + Health

• Mini agent sends heartbeat
• GPU agent sends heartbeat
• Redis stores active worker state
• Broker marks stale workers unavailable
• Simple status endpoint shows worker health

Phase 3 — Job Queue + Basic Scheduling

• Add request queue
• Add least-busy Mini selection
• Add least-busy GPU selection
• Track job state: queued → assigned → running → complete/failed
• Handle “all GPUs busy” without losing the request

Phase 4 — Sessions + Context

• Durable session exists in DB
• Working session context can be assigned to a Mini
• Project files/context can be cached on Mini
• Returning requests can reuse useful context
• Mini failure does not destroy durable session state

Phase 5 — Multi-GPU / Multi-Mini Scaling

• Add more Minis
• Add multiple GPUs
• Test independent one-GPU jobs in parallel
• Compare independent GPU scheduling vs grouped GPU execution
• Measure queueing and utilization

Phase 6 — Production / Development Separation

• Production Mini pool + Rig 1
• Development Mini pool + Rig 2
• Test promotion/failover process
• Test taking one rig offline
• Define deployment and rollback procedure

---

[Alex668866]

I'd generate one request ID at the API in phase 1 and carry it through the Mini, broker, and GPU logs. That'll separate queueing, network, and time-to-first-token delays before scheduler complexity makes the bottleneck harder to find.