Adi - Ground 0

[adikatre]

Title: Ground 0: Define the OCS Intelligence distributed AI system, inference goals, deployment path, and team execution plan

Description:

The goal is not just to “make AI work,” but to build a usable, measurable, and maintainable architecture for students — with the API already set up, and the next work focused on making the system better and better for classroom use.

---

1. Inference bottlenecks and model performance

From issue #4, the project has already identified that inference performance is a major constraint.

Problems called out

• low TPS / throughput
• low compute and GPU utilization
• high TTFT
• cold start delays
• inter-GPU bandwidth limiting performance
• context-window pressure, including the 256k context concern

Specific work to address

• Measure baseline inference performance with verbose output and recorded stats
• Track:
  • TPS / eval rate
  • TTFT
  • load duration
  • prompt eval duration
  • cold start vs warm start behavior
• Identify where bottlenecks come from:
  • GPU utilization
  • model loading
  • bandwidth between GPUs
  • cold start behavior after idle time
• Test whether speculative decoding can improve responsiveness
• Evaluate whether YaRN or similar approaches are needed for larger context behavior
• Decide which improvements are actually useful for the hardware in this repo, rather than assuming generic modern inference settings will work

Outcome expected

A documented path toward better inference performance that is based on actual measurement, not assumptions.

---

2. Deployment strategy for GitHub Copilot / Claude Code

From issue #5, deployment is a project concern, not just a technical afterthought.

What the issue represents

• A deployment strategy discussion for GitHub Copilot / Claude Code
• A sign that the project needs a clearer path from research/prototype to something students can actually use
• A reminder that deployment should support the overall system direction, not live separately from it

Specific work to address

• Define how the current project should be deployed in a way that supports the intended user workflow
• Clarify what is public-facing vs private infrastructure
• Make the deployment strategy compatible with the project’s AI / assistant use case
• Ensure deployment choices do not fight the planned architecture for:
  • cloud control plane
  • private networking
  • worker nodes
  • GPU inference
• Document how deployment decisions fit into the student-facing experience

Outcome expected

A deployment path that supports the broader OCS Intelligence architecture instead of being treated as a separate concern.

---

3. Communication, research, and sprint burndown tracking

From issue #6, the repo already shows that project organization and execution tracking are important.

Items already identified

• document the most recent group split
• finish literature review on inference engineering for low-compute use cases
• add designated research questions to the capstone
• justify experiments clearly
• update the capstone PR
• setup Cloudflare Tunnel
• solve inference bottlenecks from issue Solving Inference Bottlenecks (Bad TPS, ITL, TTFT, etc.) #4

Specific work to address

• Keep the team split visible and documented
• Tie research questions to specific engineering decisions
• Make sure experiments are justified by how they help solve the problem statement
• Track progress in a burndown-style way so work does not remain too abstract
• Ensure infrastructure work, research work, and deployment work are connected

Outcome expected

A clearer execution model where research, implementation, and documentation all point to the same project goal.

---

4. Shared project vision and architecture direction

From issue #7, the project has a much deeper architecture story than just “run AI on some machines.”

Main architectural ideas

• a public cloud control plane
• private networking through NetBird
• a Mac Mini pool for session, context, and orchestration work
• GPU rigs for inference
• a broker/scheduler between clients, Minis, and GPUs
• a split between durable state, fast state, and working state
• a two-pool model for production and development/testing

Specific work to address

• Keep the system organized around the control plane vs compute plane split
• Use the correct storage layer for the correct state:
  • durable state in RDS
  • fast live state in Redis
  • disposable working state on Minis
• Preserve the intended flow:
  • student → HTTPS/public endpoint → broker → Mini → GPU → response
• Make sure networking stays private for infrastructure while remaining simple for students
• Continue to use the architecture as a research platform for low-cost distributed AI inference

Outcome expected

A coherent system model that keeps student access simple while making the backend architecture understandable and scalable.

---

5. Team burndown and Ground Zero readiness

From issue #8, the Ground 0 work is also about helping others understand the project phase.

Relevant points

• Make a lesson to teach Mr. Mort about phase 0 / basic setup
• Everyone has to finish Ground 0

Specific work to address

• Turn the Ground 0 phase into something teachable
• Make the setup and architecture understandable to someone joining late
• Document the basic setup clearly enough that the team can align on what “finished Ground 0” means
• Use the lesson/doc approach so team members and mentors can see what was learned, not just what was installed

Outcome expected

A Ground Zero explanation that can be shared with others and used to align the whole team.

---

6. Full Ground 0 architecture and project decomposition

From issue #10, the repo has the most complete version of the Ground 0 system design.

Key architectural details to preserve

• Student-facing entry point through a web app / API
• NetBird as the private connectivity layer
• AWS / OCS EC2 as the stable public control plane
• RDS for durable state
• Redis for live state, queues, registries, and locks
• Mac Minis as session/context/agent workers
• GPU rigs as inference workers
• llama.cpp + llama-server as the baseline GPU runtime
• GTX 1070 / Pascal hardware treated as a research constraint
• broker/scheduler logic for:
  • healthy Mini detection
  • session assignment
  • Mini disappearance handling
  • GPU selection
  • queueing
  • model locality
  • GPU busy state
  • rig failover / promotion

Specific work to address

• Convert the architecture into small service contracts
• Build the smallest end-to-end proof path:
  • client → FastAPI → NetBird → Mini → GPU → llama-server → response
• Add worker registration and health checks
• Add queue and basic scheduling behavior
• Support session/context handling on Minis
• Measure latency and failure cases
• Separate production and development/test pools
• Test failover and promotion behavior between rigs

Outcome expected

A staged implementation plan that moves from architecture into working infrastructure without trying to scale too early.

---

7. Concrete Ground Zero direction for this repo

Taken together, issues #4–#10 point to a project that should focus on:

• improving inference quality and speed
• proving deployment and networking work together
• using a broker to coordinate Minis and GPUs
• tracking state correctly across durable, live, and working layers
• documenting the system so the team can execute consistently
• building toward a student-facing AI platform instead of a loose set of machines

The API is already set up. The next work is to keep improving the system so it becomes more useful, more reliable, and more targeted for students in class.

---

Expected deliverables

• Baseline inference measurements with notes on bottlenecks
• A documented deployment path
• Updated capstone / burndown documentation
• A clearer system architecture doc or implementation plan
• A smallest-possible end-to-end request path
• Worker and scheduler contracts for Minis and GPUs
• A Ground 0 summary that can be taught to the rest of the team

[adikatre]

Below is the same issue with targeted edits only. I kept the structure and most wording intact, removed the Mac Mini architecture, added the actual hardware/inference baseline, clarified the EC2/NetBird path, and made the GitHub Copilot BYOK use case explicit.

---

Ground 0: Define the OCS Intelligence distributed AI system, inference goals, deployment path, and team execution plan

The goal is not just to “make AI work,” but to build a usable, measurable, and maintainable architecture for students, with the API already set up, and the next work focused on making the system better and better for classroom use.

---

1. Inference bottlenecks and model performance

From issue #4, the project has already identified that inference performance is a major constraint.

Problems called out

• low TPS / throughput
• low compute and GPU utilization
• high TTFT
• cold start delays
• inter-GPU bandwidth limiting performance
• context-window pressure, including the 256k context concern
• uneven utilization across GPUs during inference

Current hardware and baseline

The project currently has two physical GPU rigs, each with:

• 8× NVIDIA GTX 1070
• 8 GB VRAM per GPU

The architecture should assume 1 to N GPU rigs / logical inference workers rather than being hard-coded around exactly two machines.

A physical rig may also be divided into multiple logical workers using Proxmox and allocated subsets of GPUs when certain processes or workloads benefit from separate GPU allocation.

A current Qwen3.8 27B test was run on a server with 7 GPUs. The observed inference statistics were:

• total duration: 26.253 s
• load duration: 3.217 ms
• prompt tokens: 20
• prompt evaluation duration: 1.843 s
• prompt evaluation rate: 10.85 tokens/s
• generated tokens: 291
• generation duration: 23.275 s
• generation rate: 12.50 tokens/s

A point-in-time nvidia-smi snapshot also showed that the GPUs were not all active concurrently, with low or uneven utilization across several cards despite most of the model occupying GPU memory.

This makes GPU utilization and effective use of the available 1070s a key part of issue #4.

Specific work to address

• Measure baseline inference performance with verbose output and recorded stats

• Track:

  • TPS / eval rate
  • TTFT
  • load duration
  • prompt eval duration
  • cold start vs warm start behavior
  • per-GPU utilization
  • GPU memory usage

• Identify where bottlenecks come from:

  • GPU utilization
  • model loading
  • bandwidth between GPUs
  • how work is distributed across GPUs
  • cold start behavior after idle time

• Determine why GPUs are not being utilized effectively during a request and whether work can be distributed more efficiently

• Compare performance as GPU allocation changes, including logical workers created from subsets of GPUs

• Test whether speculative decoding can improve responsiveness

• Use YaRN to support larger context windows and measure the performance impact of larger contexts

• Continue investigating the 256k context concern, particularly its impact on memory usage, prompt processing, and responsiveness

• Decide which improvements are actually useful for the GTX 1070 / Pascal hardware in this repo, rather than assuming generic modern inference settings will work

Outcome expected

A documented path toward better inference performance that is based on actual measurement, not assumptions, including a reproducible baseline against which future changes can be compared.

---

2. Deployment strategy for GitHub Copilot / Claude Code

From issue #5, deployment is a project concern, not just a technical afterthought.

What the issue represents

• A deployment strategy discussion for GitHub Copilot / Claude Code
• A sign that the project needs a clearer path from research/prototype to something students can actually use
• A reminder that deployment should support the overall system direction, not live separately from it

The primary intended student workflow is to expose OCS Intelligence as an OpenAI-compatible endpoint that can be inserted into GitHub Copilot through BYOK.

Current deployment state

• An AWS EC2 VPS acts as the public-facing system entry point / control plane

• NetBird currently connects:

  • AWS EC2 VPS → GPU Rig 1

• An inference endpoint exists

• The endpoint has not yet been fully tested or verified end-to-end

• GPU Rig 2 must fit into the same general architecture as additional rigs are added

• The system should support 1 to N GPU rigs or logical inference workers

Specific work to address

• Verify the existing endpoint end-to-end before adding unnecessary complexity

• Prove the request path:

  GitHub Copilot / client → OpenAI-compatible EC2 endpoint → scheduler/router → NetBird → GPU inference worker → response

• Define how the current project should be deployed in a way that supports the intended student workflow

• Clarify what is public-facing vs private infrastructure

• Keep GPU rigs private behind NetBird while exposing the EC2 endpoint to clients

• Make deployment compatible with:

  • GitHub Copilot BYOK
  • OpenAI-compatible requests
  • streaming responses
  • multiple GPU rigs
  • multiple logical inference workers

• Ensure deployment choices do not fight the planned architecture for:

  • cloud control plane
  • private networking
  • GPU inference
  • scheduling and routing

• Document how deployment decisions fit into the student-facing experience

Outcome expected

A deployment path that supports the broader OCS Intelligence architecture instead of being treated as a separate concern, with a verified OpenAI-compatible endpoint that can be used through GitHub Copilot BYOK.

---

3. Communication, research, and sprint burndown tracking

From issue #6, the repo already shows that project organization and execution tracking are important.

Items already identified

• document the most recent group split
• finish literature review on inference engineering for low-compute use cases
• add designated research questions to the capstone
• justify experiments clearly
• update the capstone PR
• setup and verify Cloudflare / external endpoint access as needed
• verify NetBird connectivity and the existing inference endpoint
• solve inference bottlenecks from issue Solving Inference Bottlenecks (Bad TPS, ITL, TTFT, etc.) #4

Specific work to address

• Keep the team split visible and documented
• Tie research questions to specific engineering decisions
• Make sure experiments are justified by how they help solve the problem statement
• Track progress in a burndown-style way so work does not remain too abstract
• Ensure infrastructure work, research work, and deployment work are connected
• Record baseline measurements before changing inference configuration so optimization results can be compared objectively

Outcome expected

A clearer execution model where research, implementation, and documentation all point to the same project goal.

---

4. Shared project vision and architecture direction

From issue #7, the project has a much deeper architecture story than just “run AI on some machines.”

Main architectural ideas

• a public cloud control plane on AWS EC2
• private networking through NetBird
• GPU rigs for inference
• a broker/scheduler between clients and available GPU inference workers
• support for 1 to N physical or logical GPU workers
• ability to split physical GPU rigs into multiple workers through Proxmox / GPU allocation
• a split between durable state and fast/live state
• a two-pool model for production and development/testing

Intended request flow

Student / GitHub Copilot
↓
HTTPS / OpenAI-compatible endpoint on AWS EC2
↓
Broker / scheduler
↓
NetBird private network
↓
Selected GPU rig or logical GPU worker
↓
Inference runtime
↓
Stream response back to GitHub Copilot

The AWS EC2 VPS, rather than a separate Mac Mini layer, is responsible for directing traffic to the appropriate backend resources.

Specific work to address

• Keep the system organized around the control plane vs compute plane split
• Keep infrastructure networking private while remaining simple for students
• Use the EC2 VPS as the stable routing and control point
• Allow the scheduler to treat physical rigs and Proxmox-created GPU allocations as available inference workers
• Avoid architecture that assumes one request always maps to one specific physical rig
• Continue to use the architecture as a research platform for low-cost distributed AI inference
• Preserve the ability to expand from the current two physical rigs to additional rigs later

Outcome expected

A coherent system model that keeps student access simple while making the backend architecture understandable, measurable, and expandable.

---

5. Team burndown and Ground Zero readiness

From issue #8, the Ground 0 work is also about helping others understand the project phase.

Relevant points

• Make a lesson to teach Mr. Mort about phase 0 / basic setup
• Everyone has to finish Ground 0

Specific work to address

• Turn the Ground 0 phase into something teachable

• Make the setup and architecture understandable to someone joining late

• Document the basic setup clearly enough that the team can align on what “finished Ground 0” means

• Explain:

  • EC2 control plane
  • NetBird networking
  • GPU rigs
  • logical GPU workers
  • scheduling
  • OpenAI-compatible API
  • GitHub Copilot BYOK
  • inference measurement

• Use the lesson/doc approach so team members and mentors can see what was learned, not just what was installed

Outcome expected

A Ground Zero explanation that can be shared with others and used to align the whole team.

---

6. Full Ground 0 architecture and project decomposition

From issue #10, the repo has the most complete version of the Ground 0 system design.

Key architectural details to preserve

• Student-facing entry point through an OpenAI-compatible API

• GitHub Copilot BYOK as a primary student client

• NetBird as the private connectivity layer

• AWS / OCS EC2 as the stable public control plane

• RDS for durable state

• Redis for live state, queues, registries, and locks

• GPU rigs as inference workers

• two current physical rigs with 8× GTX 1070 each

• support for 1 to N GPU rigs / inference workers

• ability to use Proxmox / allocated GPU subsets to divide a physical rig into multiple logical workers where useful

• llama.cpp + llama-server as the baseline GPU runtime

• GTX 1070 / Pascal hardware treated as a research constraint

• YaRN for larger context-window support

• broker/scheduler logic for:

  • healthy worker detection
  • worker registration
  • GPU / worker selection
  • queueing
  • model locality
  • GPU busy state
  • logical GPU allocation
  • physical rig availability
  • rig failover / promotion

Specific work to address

• Convert the architecture into small service contracts

• Build the smallest end-to-end proof path:

  GitHub Copilot/client → EC2 API → NetBird → GPU worker → inference runtime → streamed response

• Verify the currently existing EC2 → NetBird → GPU Rig 1 endpoint

• Add GPU Rig 2 using the same architecture

• Add worker registration and health checks

• Add queue and basic scheduling behavior

• Make scheduling work against a general pool of workers rather than hard-coded machine names

• Support logical workers created from allocated subsets of GPUs

• Measure latency and failure cases

• Measure inference performance across different GPU allocation strategies

• Separate production and development/test pools

• Test failover and promotion behavior between rigs

• Test larger contexts using YaRN

• Measure whether larger context support creates unacceptable prompt-processing or memory overhead

Outcome expected

A staged implementation plan that moves from architecture into working infrastructure without trying to scale too early.

The first priority is proving the smallest working path before introducing more complicated scheduling:

Copilot → EC2 → NetBird → GPU Rig 1 → model → streamed response

Once that works reliably, the architecture can expand to Rig 2 and additional logical workers.

---

7. Concrete Ground Zero direction for this repo

Taken together, issues #4–#10 point to a project that should focus on:

• improving inference quality and speed
• increasing effective utilization of the available GTX 1070 GPUs
• understanding why the current 7-GPU Qwen3.8 27B configuration produces roughly 12.5 generated tokens/s
• improving TTFT and cold-start behavior
• measuring rather than assuming the cause of multi-GPU bottlenecks
• using YaRN for larger-context support while measuring its performance cost
• proving deployment and networking work together
• verifying the existing EC2 → NetBird → GPU Rig 1 path
• exposing the system through an OpenAI-compatible endpoint for GitHub Copilot BYOK
• using the EC2 VPS as the primary traffic-routing and control layer
• using a broker to coordinate 1 to N physical or logical GPU workers
• supporting the two existing 8× GTX 1070 rigs
• allowing rigs to be subdivided with Proxmox / allocated GPUs where that improves resource usage
• tracking state correctly across durable and live layers
• documenting the system so the team can execute consistently
• building toward a student-facing AI platform instead of a loose set of machines

The API is already set up. The next work is to verify the complete request path first, establish a reproducible performance baseline, and then improve scheduling and inference so the system becomes more useful, reliable, and targeted for students in class.

---

Expected deliverables

• Verified GitHub Copilot / OpenAI-compatible endpoint → EC2 → NetBird → GPU Rig request path
• Baseline Qwen3.8 27B inference measurements
• TPS / TTFT / prompt-processing / cold-start measurements
• Per-GPU utilization measurements
• Notes identifying likely inference bottlenecks
• Comparison of GPU allocation / worker configurations where useful
• YaRN large-context testing and measurements
• A documented deployment path
• Updated capstone / burndown documentation
• A clearer system architecture doc or implementation plan
• A smallest-possible end-to-end request path
• Worker and scheduler contracts for GPU workers
• Support for 1 to N physical or logical GPU workers
• Rig health checking and basic failover behavior
• A Ground 0 summary that can be taught to the rest of the team