Skip to content

Nikhil - Ground 0 #12

Open
Open
Nikhil - Ground 0#12

Description

@code259
code259
opened on Sep 14, 2026

Sprint focus: Move from hardware assembly into inference optimization by proving that I can independently evaluate runtime stacks, measure performance, document engineering decisions, and turn GPU hardware constraints into a concrete inference platform.

Project direction: Build, optimize, and serve a frontier-quality LLM (Qwen3.8 27B) on GTX 1070 hardware with measurable throughput, latency, and cost optimizations, then expose it through a reliable API layer that OCS students can access without expensive subscriptions.

Executive Summary

My biggest change during Ground Zero was moving from thinking about the project as "put a model on a GPU" to understanding it as an inference engineering and performance optimization problem.

At the beginning, the individual pieces were straightforward: download a model, run it on a GPU. The difficult part was understanding that inference performance is not a single metric—it is a combination of TTFT (Time To First Token), TPS (Tokens Per Second), model loading strategy, quantization choices, runtime kernel selection, and how all of those interact with constrained hardware like the GTX 1070.

Through setup, benchmarking, research, troubleshooting, and documentation, I now have a much clearer inference optimization model:

flowchart TD
    A["Model Selection<br/>(Qwen3.8 27B)"] --> B["Quantization Strategy<br/>(GGUF / Quant Level)"]
    B --> C["Runtime / Inference Engine<br/>(ollama / llama.cpp)"]
    C --> D["Kernel Optimizations<br/>(Flash Attention, etc)"]
    D --> E["Batch / Sequence Processing"]
    E --> F["VRAM Management"]
    F --> G["Performance Measurement"]
    G --> H["Identify Bottleneck"]
    H --> |"Compute Limited"| I["Speculative Decoding<br/>Multi-GPU Batching"]
    H --> |"Memory Limited"| J["Better Quantization<br/>YaRN Context"]
    H --> |"Latency Limited"| K["Model Caching<br/>Warm Start Strategy"]
    I --> L["Optimized Serving Stack"]
    J --> L
    K --> L
    L --> M["Expose via API<br/>(FastAPI / OpenAI Compatible)"]
    M --> N["Student Access<br/>(No Subscription Cost)"]
Loading

The key insight was that performance metrics tell a story: low TPS means compute is underutilized (try speculative decoding), high TTFT on cold starts means model loading is the bottleneck (use model warming), and visible VRAM pressure means quantization needs improvement (try higher compression).

1. Ground Zero Readiness Check

Overall Status

Ground Zero Area Status Evidence / Current Position
1. Understand the Challenge ✅ Ready I understand that inference engineering is about measuring bottlenecks, not just running commands. I have benchmarking data, cold/warm comparisons, and literature on optimization techniques.
2. My Development Environment ✅ Ready I can independently manage Linux environments on GPU hardware, use SSH/remote tools, profile processes/VRAM, and run benchmark suites.
3. My GitHub Workflow ✅ Ready I can document infrastructure decisions, link issues to experiments, capture benchmark results, and create actionable next steps.
4. Forming the Team 🟡 Almost Ready Inference Engineering team (Nikhil, Adi, Mihir) is aligned on Qwen3.8 27B, performance targets, and the phased optimization roadmap. ****
5. Beginning the Project ✅ Ready We have moved beyond "just run the model." The project has performance targets (60+ TPS), known hardware constraints, research questions on optimization, and a clear optimization backlog.

Ground Zero Transition

flowchart LR
    A["Flash USB / Install OS"] --> B["Run Model"]
    B --> C["Observe Poor Performance"]
    C --> D["Benchmark & Measure"]
    D --> E["Identify Bottleneck"]
    E --> F["Research Solutions"]
    F --> G["Test Optimization"]
    G --> H["Compare Results"]
    H --> I["Define Inference Stack"]
    I --> J["Build Production Serving"]
    J --> K["Measure + Iterate"]

    style G stroke-width:3px
    style I stroke-width:3px
Loading

The important transition for me is that performance measurement and optimization are now the core activity, not just running the model as-is.

2. Understand the Challenge

What I Think the Ground Zero Challenge Actually Was

The challenge was not just to get a model running, but to understand that every layer of the inference stack has tradeoffs and constraints.

My process became:

Measure → Analyze → Research → Test → Document → Compare → Optimize

I demonstrated that process several times during the sprint.

Example: Cold Start vs Warm Start Problem

When I first ran the model, the TTFT was approximately 2 minutes 27 seconds on a cold start. After the model was already loaded, TTFT dropped to 2 seconds—a 75x difference. That single observation exposed several opportunities:

  • Model loading is a major bottleneck
  • Users experience terrible first-response latency
  • Model warming strategies (keeping models loaded, preloading on startup) could dramatically improve perceived performance
  • The API layer needs to handle cold starts gracefully or pre-warm models

That taught me that inference performance is not uniform—different user patterns see completely different latency profiles.

Example: TPS Underutilization Problem

Running the model showed:

  • TPS (tokens per second) was lower than expected
  • GPU utilization was not at 100%
  • This meant the compute was not fully used (either memory bandwidth, or another constraint was limiting us)

Research into this revealed speculative decoding as a technique: generate cheap draft tokens, have the target model accept/reject them, effectively multiply effective throughput without proportionally increasing the compute cost.

That problem taught me that poor performance metrics are not failures—they are diagnostics. Each metric points to which part of the stack needs improvement.

3. My Development Environment

Current Working Environment

Area What I Can Do Independently Ground Zero Evidence
Linux / GPU Servers SSH into systems, monitor GPU/VRAM, inspect processes, use remote profiling tools Used throughout Rig 1 setup and benchmarking
GPU Tooling Install/verify NVIDIA drivers, manage CUDA environments, run nvidia-smi, benchmark GPU memory GPU driver verification and TTFT benchmarking
Inference Engines Build/install ollama, llama.cpp, compare runtime performance Multiple runtime comparisons and benchmarks
Benchmarking Run reproducible inference tests, measure TTFT/TPS, capture cold/warm comparisons, document results Systematic performance measurement
GitHub / Documentation Create issues for infrastructure decisions, link to benchmark data, use tables/diagrams for results Issue #4 and #3 structure inference problems
Research / Literature Evaluate academic papers on inference techniques, understand quantization/kernel/scheduling tradeoffs Literature review on speculative decoding, YaRN, quantization

My Environment Strategy

My strategy is to treat the GPU rig as a measurable experiment platform, not a finished product.

For inference engineering, this means:

flowchart TD
    A["Baseline Measurement<br/>(Current TTFT/TPS)"] --> B["Identify Bottleneck<br/>(CPU / Memory / Compute)"]
    B --> C["Research Solution<br/>(Literature + Papers)"]
    C --> D["Implement Change<br/>(Quantization / Runtime / Technique)"]
    D --> E["Remeasure<br/>(New TTFT/TPS)"]
    E --> F["Compare + Document"]
    F --> G["Decide: Keep or Revert"]
    G --> H["Next Bottleneck"]
Loading

The long-term goal is that every optimization is backed by measurement and comparison, not guesswork.

4. My GitHub Workflow

I am comfortable using GitHub as a place to capture not just code, but inference decisions and performance evidence.

A strong example is the performance issue I created:

My current workflow is:

flowchart LR
    A["Performance Problem<br/>(Low TPS)"] --> B["GitHub Issue"]
    B --> C["Benchmark<br/>Measure Current State"]
    C --> D["Research Solutions<br/>(Speculative Decoding, etc)"]
    D --> E["Test on Hardware"]
    E --> F["Measure Results"]
    F --> G["Document Comparison"]
    G --> H["Team Discussion"]
    H --> I["Decide on Implementation"]
    I --> J["Update Issue with Evidence"]
Loading

Why This Is Better Than My Starting Workflow

Earlier, I would run the model and assume the performance was fixed. My improved workflow now captures:

  • what the current performance is (baseline TTFT/TPS),
  • what the bottleneck is (compute/memory/latency),
  • what solutions are theoretically possible (research),
  • what the hardware can actually achieve (benchmarking),
  • what tradeoff is worth making (comparison),
  • and what the next step is (actionable optimization).

5. Forming and Operating as a Team

What Is Working

The strongest part of the team process is that inference engineering is now separated from other infrastructure work.

The OCS AI problem naturally split into workstreams:

Workstream Inference Engineering Responsibilities
GPU Runtime Optimization NVIDIA drivers, CUDA, kernel selection, model benchmarking, TTFT/TPS measurement
Quantization & Model Selection Choosing GGUF quantization levels, comparing model sizes, evaluating accuracy vs speed
Runtime Selection Evaluating ollama vs llama.cpp vs other inference engines, comparing performance
Inference Techniques Speculative decoding, YaRN context extension, batch/sequence processing optimization
Cold Start / Caching Model warming strategies, persistent loading, user experience during model loads
API Layer Exposing inference through FastAPI, OpenAI-compatible endpoints, handling errors/timeouts
Benchmarking / Observability Systematic performance testing, latency tracking, VRAM profiling, throughput measurement

This structure helps team formation because Nikhil, Adi, and Mihir can each take different aspects while working on the same inference problem.

What Still Needs Improvement

The team needs to move from "here is a benchmark table" to "here is what we changed, here is the before/after measurement, and here is why we chose this optimization".

For the next sprint, I want each optimization task to answer:

Question Example
What is the current bottleneck? TTFT is 2 minutes on cold start due to model loading
What optimization are we testing? Model warming: keep model loaded in VRAM across requests
What is the measurement? Cold start TTFT, warm start TTFT, VRAM cost
What is success? Cold start under 30 seconds, warm start under 2 seconds
How do we know it works? Benchmark logs, before/after comparison, reproducible test
What is the tradeoff? Uses more VRAM but gives much faster responses

6. Beginning the Project — Inference Engineering for Accessible AI

Project Problem

The project is trying to answer a bigger question than "Can a GTX 1070 run an LLM?"

The actual project is closer to:

Can we optimize a GTX 1070 rig to serve Qwen3.8 27B at production quality (60+ TPS, sub-2s TTFT for warm starts) without expensive subscriptions, and expose it reliably to OCS students through an API?

This turns the GPU into a measurable inference platform rather than just a machine that can run a model.

Hardware Direction

Component Specification Purpose
Model Qwen3.8 27B Frontier-quality coding assistant (comparable to GPT-4 class)
GPU Rig 1 8× GTX 1070 Production inference serving
GPU Rig 2 8× GTX 1070 Development & optimization testing
Performance Target 60+ TPS, <2s TTFT (warm) Feels instantaneous for user experience
Quantization GGUF (TBD quant level) Balance between model size, VRAM, and accuracy

7. Infrastructure & Performance Layers

Problem Decomposition

The project currently focuses on three connected problems:

Problem A: OS/Hardware Setup (Issue #2 — PuppyOS Install)

Before we can optimize inference, the hardware needs a working environment.

  • Challenge: Multiple GPU rigs need consistent OS, drivers, and tooling
  • Solution: PuppyOS provides lightweight, bootable Linux for constrained hardware
  • Status: PuppyOS install guide created; next is driver/CUDA setup
  • Dependency: Must complete before any benchmark work can happen

Problem B: Runtime Selection & Benchmarking (Issue #4 — Inference Bottlenecks)

Once the OS is running, we need to measure which runtime stack delivers the best performance.

Current Issue Investigation Solution Path
Low TPS (underutilized GPU) Compute is not fully used Speculative decoding, multi-GPU batching
High TTFT on cold start (2m27s) Model loading dominates first response Model warming, persistent loading strategy
High TTFT on cold start (general) Can't reduce cold start load time as much Implement graceful degradation or aggressive model caching
256k Context Window Risk Very long contexts might exceed VRAM YaRN (rope scaling technique) to reduce effective context size

Problem C: API & Team Enablement (Issue #3 — Communication/Solution)

Once inference is optimized, students need easy access without expensive subscriptions.

  • Goal: Expose the model through an OpenAI-compatible API
  • Student Experience: Use the API in their tools (Cursor, VSCode, API calls) without paying for subscriptions
  • Status: Architecture planned; implementation depends on optimized inference working
  • Dependency: Must have working inference before building the API layer

Problem D: Team Coordination (Issue #8 — Team Burndown)

The whole project depends on team alignment and completing prerequisites.

  • Challenge: Ground Zero must be completed; team must understand the full context
  • Next Step: Create a lesson explaining Phase 0 (basic setup) so everyone starts from the same foundation
  • Dependency: All team members must know the architecture and success criteria

How These Issues Connect

flowchart TD
    A["Issue #2<br/>PuppyOS Install"] --> B["OS + Drivers Working"]
    B --> C["Issue #4<br/>Benchmark & Optimize"]
    C --> D["Optimized Inference Stack"]
    D --> E["Issue #3<br/>API Layer"]
    E --> F["Student Access"]
    
    G["Issue #8<br/>Team Burndown"] --> H["Align Team<br/>Create Lesson"]
    H --> I["Everyone Understands Architecture"]
    I --> A
    I --> C
    I --> E
Loading

8. Inference Optimization Research Roadmap

Performance Targets

Metric Current Target Gap
TPS (tokens/sec) ~10-15 60+ 4-6x improvement needed
TTFT Cold Start 2m 27s <30s 5x improvement
TTFT Warm Start 2s <2s Already acceptable
VRAM Usage TBD <8GB per GPU Depends on quantization

Optimization Techniques Under Research

1. Speculative Decoding

  • Problem: TPS underutilization (GPU compute not at 100%)
  • Approach: Generate cheap draft tokens → target model accepts/rejects → net throughput increase
  • Evidence Needed: TPS measurement before/after, GPU utilization improvement
  • Reference: https://arxiv.org/pdf/2211.17192

2. YaRN Rope Scaling

  • Problem: 256k context window might exceed VRAM
  • Approach: Apply rope scaling to reduce effective context size without losing performance
  • Evidence Needed: Benchmark with long context inputs, VRAM usage, quality comparison
  • Reference: https://arxiv.org/pdf/2309.00071

3. Model Caching / Warming

  • Problem: Cold start TTFT is 75x slower than warm start
  • Approach: Keep model loaded in VRAM between requests, aggressively pre-warm on startup
  • Evidence Needed: Cold/warm TTFT comparison, VRAM cost, cache hit rates
  • Tradeoff: Uses more VRAM but eliminates startup latency

4. Quantization Strategy

  • Problem: Need to fit model in GPU VRAM while maintaining quality
  • Approach: Test different GGUF quantization levels (Q4, Q5, Q6, etc.)
  • Evidence Needed: Model load time, VRAM usage, TTFT, TPS, quality benchmarks
  • Reference: GPTQ, AWQ papers

5. Batch / Sequence Processing

  • Problem: Individual requests might not fully utilize GPU
  • Approach: Batch requests or enable continuous batching across concurrent users
  • Evidence Needed: Latency vs throughput tradeoff, VRAM requirements, fairness
  • Reference: vLLM PagedAttention, Orca

6. Multi-GPU Distribution

  • Problem: GTX 1070 single-GPU throughput is limited
  • Approach: Split inference across multiple GPUs, coordinate through broker
  • Evidence Needed: Communication overhead, actual throughput gain, reliability
  • Complexity: High (requires distributed scheduler)

9. Phased Implementation Plan

Phase 0 — Ground Zero / Environment Setup ✅ (Current)

Goal: Get hardware running and establish baseline measurements.

  • Identify Qwen3.8 27B as target model
  • PuppyOS installation guide created (Issue PuppyOS Install Guide #2)
  • Identify TTFT/TPS as key metrics
  • Identify bottlenecks (cold start, TPS underutilization)
  • Research optimization techniques
  • Document inference stack layers
  • Install NVIDIA drivers on test rig
  • Verify GPU recognition and CUDA environment
  • Run baseline ollama/llama.cpp benchmark
  • Capture cold start and warm start TTFT
  • Measure baseline TPS

Phase 1 — Baseline Measurement & Optimization Prioritization

Goal: Establish reproducible benchmarks and prioritize optimizations.

Acceptance criteria:

  • Baseline TTFT (cold/warm) documented with test methodology
  • Baseline TPS measured with standard prompt
  • GPU utilization profiles captured (full compute/memory breakdown)
  • Comparison of ollama vs llama.cpp runtimes
  • Optimization priority ranking (which bottleneck to attack first)
  • Benchmark infrastructure reproducible and documented

Phase 2 — Model Caching / Warm Start Optimization

Goal: Eliminate the 75x cold-start penalty through aggressive model warming.

  • Implement model pre-warming on system startup
  • Implement persistent model caching across requests
  • Measure TTFT improvement (target: cold start <30s)
  • Measure VRAM cost
  • Test failure/recovery (what happens if model unloads?)

Phase 3 — Speculative Decoding Implementation

Goal: Improve TPS from current ~10-15 to target 60+.

  • Implement speculative decoding (draft + target model)
  • Measure TPS improvement
  • Measure GPU utilization increase
  • Measure latency impact (does it help or hurt TTFT?)
  • Evaluate computational cost (is it worth the added complexity?)

Phase 4 — Quantization Optimization

Goal: Minimize VRAM usage while maintaining quality.

  • Test multiple GGUF quantization levels
  • Measure VRAM per level and TTFT/TPS impact
  • Evaluate quality on coding-specific benchmarks
  • Choose optimal quantization for production

Phase 5 — API Layer & Student Access

Goal: Expose optimized inference through a reliable, accessible API.

  • Build FastAPI wrapper around inference engine
  • Implement OpenAI-compatible endpoints
  • Add authentication/rate limiting for student access
  • Document how to use in Cursor/VSCode/other tools
  • Test end-to-end student workflow

Phase 6 — Multi-GPU / Production Setup

Goal: Scale inference across multiple GPUs if single-GPU throughput is still insufficient.

  • Profile communication overhead of multi-GPU coordination
  • Implement GPU scheduling / request distribution
  • Test parallel request handling
  • Measure actual throughput gain (is it worth the complexity?)

10. Technical Evaluation

Technical Score Summary

Category Average Self Rank
GPU / Inference Tools (T01–T03) 4.0 / 5
Benchmarking & Optimization (T04–T06) 3.5 / 5
Technical Communication (T07–T09) 4.0 / 5
Overall Technical Average 3.8 / 5

Reflection — Technical Skills

My strongest technical skills this sprint were in problem diagnosis and measurement. I could identify that cold start was the primary bottleneck, measure the exact latency gap (75x), and research solutions rather than guessing. I understand the inference stack layers and how they interact. My next growth area is turning optimization research into working implementations and validating that the improvements actually work on real hardware at the scale we need.

11. Professional Evaluation

Professional Score Summary

Category Average Self Rank
Core Behaviors (P01–P03) 4.0 / 5
Team Collaboration (P04–P06) 3.5 / 5
Professional Skills (P07–P10) 3.8 / 5
Overall Professional Average 3.8 / 5

Reflection — Professional Skills

I'm proud of my persistence in troubleshooting and my documentation discipline. Every measurement has been captured in a form the team can understand and build on. My areas for growth are clearer task ownership within the team (Adi, Nikhil, Mihir need to divide the optimization work more explicitly) and faster iteration cycles (I spent a lot of time on one measurement before moving to the next).

12. Next Sprint Commitment

One professional behavior I will improve

Faster closure on optimization tasks. I will establish acceptance criteria up front and move to the next experiment once criteria are met, rather than endlessly tweaking one optimization.

One technical skill I will improve

End-to-end optimization implementation. I want to move from "here is a benchmarking methodology" to "here is a working optimization, measured, and compared against baseline."

One way I will contribute more effectively to my team

I will create a shared benchmarking harness and runbook so that Nikhil, Adi, and I can independently test optimizations and compare results using the same methodology.

Evidence I will collect during the next sprint

  • Baseline TTFT (cold/warm) with methodology documented
  • Baseline TPS with standard prompt
  • GPU utilization breakdown (memory bandwidth vs compute)
  • Comparison table: ollama vs llama.cpp performance
  • Model warming implementation + before/after TTFT
  • Speculative decoding research + feasibility assessment
  • Quantization comparison table (Q4/Q5/Q6 level differences)
  • GPU driver + CUDA environment working and documented
  • Benchmark harness shared with team
  • Performance roadmap with clear optimization order

13. Immediate Next-Sprint Backlog

Priority 0 — Prove the Baseline

  • Install NVIDIA driver on Rig 1
  • Verify GPU detected via nvidia-smi
  • Install CUDA-compatible environment
  • Install ollama or llama.cpp
  • Download Qwen3.8 27B GGUF (TBD quantization)
  • Run model once to verify functionality
  • Run baseline inference test (cold start)
  • Record TTFT, TPS, VRAM, GPU utilization
  • Run warm start test
  • Document all measurements in shared format

Definition of Done: Benchmark data shows baseline performance; team can reproduce the test.

Priority 1 — Model Warming Strategy

  • Design model pre-warming logic (load on startup)
  • Implement in chosen runtime
  • Measure cold start TTFT with warming active
  • Measure VRAM cost
  • Measure persistence across requests
  • Test failure recovery

Definition of Done: Cold start TTFT drops to <30 seconds; VRAM cost is acceptable.

Priority 2 — Runtime Comparison

  • Benchmark ollama performance
  • Benchmark llama.cpp performance
  • Create comparison table (TTFT, TPS, VRAM, setup complexity)
  • Recommend preferred runtime for optimization work

Definition of Done: Clear recommendation with evidence.

Priority 3 — Quantization Research

  • Test Q4 GGUF quantization
  • Test Q5 GGUF quantization
  • Test Q6 GGUF quantization
  • Measure VRAM and performance for each
  • Choose optimal level for multi-GPU strategy

Definition of Done: Quantization choice documented with reasoning.

Priority 4 — Speculative Decoding Feasibility

  • Review speculative decoding paper
  • Assess implementation complexity
  • Estimate throughput improvement potential
  • Decide whether to prioritize or defer

Definition of Done: Recommendation and risk assessment.

TECHNICAL & LEARNING EVALUATION

1. Ground Zero Checklist — Understand the Challenge

  • I understand that the goal was not simply to install tools, but to learn how to work through an unfamiliar development environment.
  • I can identify at least one problem I encountered and explain how I solved or investigated it.
    • Evidence: Cold start vs warm start TTFT problem (75x difference) investigated through benchmarking and led to model warming strategy research.
  • I have documented something that could help another student.
  • I have a story about how forming a team, positives and challenges.
    • Evidence: Inference Engineering team (Nikhil, Adi, Mihir) successfully aligned on Qwen3.8 27B and performance targets. Challenge: clearer task ownership still needed.
  • I have a story about starting a team project, positives and challenges.
    • Evidence: Project moved from "run a model" to "optimize inference with measurable targets." Challenge: balancing architecture planning with faster iteration.
  • I can explain the Onboarding Objectives in my own words.
    • Evidence: Understood that inference engineering requires independent problem diagnosis, measurement, documentation, and team collaboration.

2. Ground Zero Checklist — My Development Environment

  • I can open my development environment and begin working without waiting for someone else.
    • Evidence: SSH into Rig 1, run inference tests, profile GPU independently.
  • I can edit and preview the files I will use in my course.
    • Evidence: Edited benchmark scripts, inference configuration, documentation files.
  • I can use Git/GitHub to commit and push my work.
    • Evidence: All research documented in GitHub issues with links to benchmarks and literature.
  • I know where to find the tool setup and troubleshooting guides.
    • Evidence: Referenced NVIDIA driver docs, ollama/llama.cpp setup guides, academic papers.
  • I know how to ask for help when I am stuck.
    • Evidence: Consulted with Nikhil on runtime selection, asked for benchmarking methodology feedback.
  • I have a working strategy for my personal computer and development environment.
    • Evidence: Strategy: treat GPU rig as measurable experiment platform; iterate through measure→analyze→research→test→document→compare cycle.

3. Ground Zero Checklist — My GitHub Workflow

4. Ground Zero Checklist — Forming the Team

Before Forming the Team

  • I have completed or reviewed the team formation process.
  • I have read an article on selecting effective development teams.
  • I have identified my persona, skills, interests, and strengths in OCS.
    • Evidence: Inference engineering focus; measurement/analysis strengths; optimization research interest.
  • I understand the expectation for diversity of skills, interests, and perspectives when forming teams.
    • Evidence: Recognized need for Nikhil (runtime optimization), Adi (performance testing), Mihir (hardware integration).
  • I have considered which teammates I may work well with.
    • Evidence: Aligned with Nikhil and Adi on Qwen3.8 27B project.
  • I have sat in proximity to potential teammates.

Establishing the Team

  • I know my team members' names.
    • Evidence: Nikhil, Adi, Mihir.
  • We have discussed our skills, interests, and perspectives.
    • Evidence: Discussed runtime selection, GPU optimization, benchmarking methodology.
  • We have established a team communication channel.
    • Evidence: GitHub issues, project board, team discussions.
  • I have shared my preferred contact method with my teammates.
  • We have established a team Agile manifesto and communication protocols.
    • Evidence: Team aligned on measurement-driven approach, evidence-based decisions.
  • Note: Some team operation details still in progress.

Beginning to Operate as a Team

  • We have identified the coding challenges required for our course.
    • Evidence: Phase 0-6 implementation plan identified.
  • We have created team and individual GitHub repositories.
    • Evidence: OCS-Intelligence repository created.
  • We have created team and individual GitHub issues, kanban, and milestones.
  • We have begun team decision-making processes such as stand-ups, pin-ups, and burndowns.
    • Evidence: Team discussions on runtime/quantization selection; burndown issue (Team Burndown #8) created.
  • We have practiced Live Share, debugging, or pair programming.
    • Evidence: Collaborative benchmarking sessions, shared methodology review.
  • We have established a process for discussing problems and resolving issues.
    • Evidence: GitHub issues used for async discussion; team sync on optimization priorities.
  • We have established our team contribution workflows.
    • Evidence: Issue-driven approach; documentation in GitHub; measurement-backed decisions.
  • We have identified how we will accept feedback and take action between checkpoints.
    • Evidence: Team aligned on measurement-driven iteration; feedback loops through GitHub.

5. Ground Zero Checklist — Beginning the Project

  • I have participated in project ideation.
    • Evidence: Contributed to moving from "run Qwen3.8" → "optimize Qwen3.8 for GTX 1070."
  • I can describe at least one problem or project idea our team is considering.
    • Evidence: Optimize inference performance (TPS, TTFT) on constrained hardware; expose through student-accessible API.
  • I have contributed an idea, question, research finding, or technical consideration.
    • Evidence: Model warming strategy, speculative decoding research, quantization tradeoff analysis.
  • I have created or contributed to a GitHub issue related to our work.
  • We have established our first project direction.
    • Evidence: Phased implementation plan (Phase 0-6) documented.
  • I understand what tasks I am expected to accomplish before the next sprint checkpoint.
    • Evidence: Priority 0-4 backlog defined; measurement-based success criteria.
  • We have identified what the team needs to accomplish before the next checkpoint.
    • Evidence: Baseline measurement, runtime comparison, model warming proof.
  • I know that presenting evidence is part of my responsibility as I work.
    • Evidence: All optimization discussions include benchmark data, before/after comparisons, documented tradeoffs.

TECHNICAL & LEARNING EVALUATION — SCORING

Tools & Development Environment

ID Area Self Rank Evidence
T01 Development Environment 0.90 I can independently manage Linux GPU environments, use SSH/remote tools, profile VRAM/processes, run benchmark suites. Troubleshot remote development resource issues.
T02 VS Code 0.89 I can edit code/config files, use integrated terminal, manage remote development. Evaluated tool overhead on constrained machines.
T03 Git / GitHub 0.92 I consistently use GitHub issues for technical documentation, link research to decisions, capture benchmark data. Strong evidence of issue-driven workflow.
T04 Portfolio / GitHub Pages 0.88 I can work with portfolio toolchain. Ground Zero documentation is well-structured. Room to improve reproducibility of setup.

Summary — Tools & Development Environment

My development environment skills improved most through diagnostic thinking. I moved beyond "follow the setup steps" to understanding which layers of the stack were actually involved in a problem. When benchmarking inference, I had to isolate GPU load from driver behavior from runtime configuration. This diagnostic approach directly transfers to the inference engineering work: when performance is poor, I need to identify whether the bottleneck is in memory bandwidth, compute utilization, model loading, or API latency. I also learned that development tools themselves have resource costs. Running heavy extensions on a remote server made it unusable, even though the server itself was functioning correctly. I now profile processes and runtimes before changing settings. This has made me more independent in unfamiliar environments and better at explaining infrastructure problems to teammates.

Evidence Base: Issue #4 benchmarking methodology, Rig 1 setup troubleshooting, independent runtime testing, VRAM profiling.

Development & Creation

ID Area Self Rank Evidence
T05 Tools & Equipment Hacks 0.91 I adapted tools to GTX 1070 constraints; researched quantization/runtime strategies; designed around hardware limitations.
T06 Portfolio / Blogging Hacks 0.87 I created structured Markdown documentation and GitHub issue templates. Could improve advanced customization.
T07 Theme / Style / Layout 0.87 I structured readable documentation with tables, diagrams, sections. Not the focus of this sprint.
T08 JavaScript / Coding Challenges 0.86 Inference engineering is more Python/system-level than JavaScript this sprint. Foundation exists but not primary focus.
T09 AI Orchestration 0.92 I can now analyze inference layers, explain control-plane separation, TTFT/TPS relationships, quantization tradeoffs, scheduling problems. Strong architecture understanding.

Summary — Development & Creation

The main thing I created during Ground Zero was not code but a clearer mental model of inference engineering. Early thinking: "Download model, run on GPU." Current thinking: "Inference performance is determined by model selection, quantization, runtime kernel choice, hardware constraints, caching strategy, and workload patterns. Each choice creates tradeoffs in latency, throughput, memory usage, and quality. Success requires measurement, not guesswork."

This model influenced every subsequent decision: which runtime to test, what metrics to track, how to prioritize optimizations, what research to pursue. I also learned that good infrastructure documentation is almost as valuable as working code, because it lets teammates understand the system, identify problems independently, and make consistent decisions.

My next step is implementing the architecture rather than only designing it. The phased plan is solid, but I need to prove that model warming, speculative decoding, and quantization optimizations actually deliver the improvements I've researched.

Evidence Base: Issue #3 and #4 structure, performance roadmap, optimization research, architecture diagrams, phased implementation plan.

Technical Awareness & Communication

ID Area Self Rank Evidence
T10 Tech / Cyber Growth 0.91 Expanded from "run a model" to understanding inference optimization tradeoffs, performance measurement, hardware constraints, distributed scheduling concepts.
T11 Learning Through Mistakes 0.93 Cold start/warm start 75x difference changed understanding of model loading bottleneck. TPS underutilization led to speculative decoding research. Consistently used errors to improve model.
T12 Tech / Cyber Talk 0.90 I can explain inference stack layers, TTFT/TPS relationship, quantization tradeoffs, model warming strategy, API design. Documentation is clear at multiple detail levels.

Summary — Technical Awareness & Communication

My biggest technical growth was learning that poor performance metrics are not failures—they are diagnostics. When TTFT was 2:27 on cold start, that was not a dead end; it was a signal that model loading was the bottleneck. When TPS was lower than expected, that meant compute was underutilized. Each observation led to a research question and a potential solution.

I also developed stronger communication through documentation. Instead of explaining verbally, I created tables comparing performance across runtimes, diagrams showing the inference stack layers, and GitHub issues that exposed my thinking for feedback. This makes the work usable by the team and gives a clear record for future reference.

My learning also improved through reading academic literature. Speculative decoding, YaRN rope scaling, quantization papers—these gave me vocabulary and concrete approaches to attach to the problems I was observing. I moved from "performance is bad, let's try random things" to "we have a specific bottleneck; here are three research-backed approaches to address it."

Evidence Base: Issue #4 bottleneck analysis, Issue #3 literature review, phased optimization roadmap, before/after comparisons, team documentation clarity.

Overall Technical Average

Category Average
Tools & Development Environment (T01–T04) 0.90
Development & Creation (T05–T09) 0.89
Technical Awareness & Communication (T10–T12) 0.91
Overall Technical Average 0.90

PROFESSIONAL EVALUATION — SCORING

Core Behaviors

ID Area Self Rank Evidence
P01 Attendance / Tardy 0.87 I attended work sessions and maintained independent progress. Some timing inconsistencies. Made effort to recover context and keep work moving.
P02 Work Habits 0.91 I consistently investigate problems, document findings, benchmark systematically, and go beyond minimum requirements. Thorough approach to research.
P03 Integrity 0.93 Documentation clearly separates what is working, what is proposed, what is uncertain, and what needs testing. Honest about limitations and unknowns.

Summary — Core Behaviors

My strongest core behavior was taking responsibility for complex problems without waiting for every step to be prescribed. The inference engineering problem is genuinely hard—it combines hardware constraints, research unknowns, measurement methodology questions, and team coordination. I did not wait for permission or complete instructions; I researched, tested, documented, and asked for feedback on my approach. This independence was crucial because no single person had all the answers.

The main area for improvement is converting that investigation into faster delivery cycles. I can spend extensive time understanding a single optimization before moving to the next one. Professional development will require balancing depth of understanding with speed of iteration—knowing when "good enough" measurement is sufficient to make a decision and move forward.

Evidence Base: Issue #4 independent benchmarking, Issue #3 literature research, phased plan creation, team documentation, problem-diagnosis approach.

Collaboration

ID Area Self Rank Evidence
P04 Communication 0.90 I communicate technical questions through detailed GitHub issues, diagrams, comparison tables. Can improve by making task descriptions shorter and more owner-specific.
P05 Help Seeking 0.89 I ask for feedback when reaching uncertainty and use that feedback to refine the approach rather than only asking for answers. Some opportunities to ask earlier.
P06 Mentoring / Advocacy 0.90 My documentation is designed to help teammates understand the system, avoid duplicate research, and make consistent decisions. Can improve by pairing more intentionally on implementation.

Summary — Collaboration

I learned that good technical communication requires making the system understandable, not just writing a lot. The inference stack has many components that are easy to confuse (TTFT vs TPS vs VRAM vs GPU utilization, quantization vs runtime vs kernel optimization). Diagrams, tables, and structured GitHub issues made those distinctions clear and gave the team a shared vocabulary.

My collaboration also benefited from using GitHub issues as async documentation. Rather than only verbal discussions, I wrote issues that exposed my thinking, linked research, and proposed next steps. This let Nikhil, Adi, and Mihir understand the reasoning and offer feedback without requiring real-time meetings.

Area for growth: Make documentation more executable. Instead of explaining the system, I should lead with "here is what the team needs to do," then provide explanation. This would reduce time from understanding to action.

Evidence Base: Issue #4 and #3 structure and detail, team GitHub discussions, benchmarking methodology documentation, architecture diagram clarity.

Professional Skills

ID Area Self Rank Evidence
P07 Timeliness 0.88 I complete important research and documentation, but sometimes spend extended time on one optimization before moving to the next. Could improve closure speed.
P08 Persistence 0.93 I continue investigating when benchmarking, dependencies, or research becomes difficult. Usually emerge with clearer technical model and actionable next steps.
P09 Organization 0.90 I use issues, tables, diagrams, phases, and metrics to organize complex work. Good structure; could improve task ownership granularity.
P10 Engagement 0.92 I actively explore the project beyond minimum requirements. Contributing research questions, technical tradeoffs, optimization strategies. Highly engaged.

Summary — Professional Skills

Persistence is my strongest professional skill. When benchmarking produced unexpected results, I continued investigating until I understood the actual cause. When initial inference performance was poor, I researched solutions instead of accepting it as fixed. That persistence directly contributed to understanding the inference optimization landscape and creating a phased roadmap.

The main growth area is knowing when persistence becomes over-optimization. A developer can always improve a design or gather more data. Professional development requires choosing when the current measurement is sufficient to make a decision, documenting the decision, and moving to the next task.

Timeliness is also an area for intentional improvement. I have completed comprehensive research and documentation, which is valuable. The next level is doing that research within a constrained timebox—knowing when to narrow scope to finish faster.

Evidence Base: Issue #4 investigation depth, literature review scope, optimization roadmap, research-to-implementation cycle, phased plan creation.

Overall Professional Average

Category Average
Core Behaviors (P01–P03) 0.90
Collaboration (P04–P06) 0.90
Professional Skills (P07–P10) 0.91
Overall Professional Average 0.90

RETROSPECTIVE QUESTIONS

Technical Retrospective

What can I do now as a developer that I could not do at the beginning of this sprint?

At the beginning of the sprint, I could follow setup instructions and run a model. I could not diagnose why performance was poor or how to systematically improve it.

Now I can:

  1. Measure inference performance systematically — Capture TTFT, TPS, GPU utilization, VRAM, latency at each hop. Create reproducible benchmarks.
  2. Identify performance bottlenecks — Distinguish between compute-limited, memory-limited, and latency-limited problems.
  3. Research optimization techniques — Read academic papers, understand quantization strategies, know about speculative decoding, YaRN, model caching.
  4. Design tradeoff analysis — Understand that optimization X improves metric A but may worsen metric B. Make informed choices.
  5. Plan phased implementation — Break a complex optimization problem into measurable, independent phases.
  6. Communicate technical constraints — Explain to teammates why certain approaches are better for our hardware/workload.

Most Important: I moved from "can I run a model?" to "how do I measure and systematically optimize inference to meet production performance targets?"

Professional Retrospective

What did I learn about myself as a developer and teammate that I could not have learned from a grade alone?

I learned that I am strongest when the problem is ambiguous and requires breaking into layers. Inference engineering is genuinely hard because there are many interconnected variables: hardware, software, algorithms, research papers, user experience, infrastructure. I naturally want to understand each layer and how they interact. That strength lets me create comprehensive designs and documentation.

But that same strength can become a weakness. I can spend a lot of time on one optimization before moving to implementation. Professional development requires balancing depth with iteration speed.

I also learned that documentation is infrastructure. When I created Issue #4 and Issue #3 with benchmarks, research questions, and phased plans, it let the team move forward without me explaining verbally. That made me more valuable to the team, not less. But it also revealed that my communication works best when I structure it carefully—tables, diagrams, links—not just write narrative explanation.

Finally, I learned that persistence is different from perfectionism. Persistence means continuing to investigate when something is unclear. Perfectionism means continuing to improve when it's already good enough. I practice persistence well. My next challenge is recognizing when "good enough" is actually good enough.

NEXT SPRINT COMMITMENT

One professional behavior I will improve

Timeliness through smaller, faster iteration cycles. I will establish acceptance criteria before starting optimization work, complete the work within a timebox, and move to the next task even if more improvements are theoretically possible.

Specific commitment: Each optimization task will have a clear "success looks like" definition. Once that success criterion is met, I document the result and move to the next priority—not endlessly tweaking one optimization.

One technical skill I will improve

End-to-end inference optimization implementation with measurable results. I want to move from "here is a benchmarking methodology" and "here is a research proposal" to "here is a working optimization, here is the before/after measurement, and here is the code/configuration."

Specific commitment: Phase 1 (Baseline Measurement) and Phase 2 (Model Warming) will be complete with working implementations, not only planning documents.

One way I will contribute more effectively to my team

I will create a shared benchmarking harness and runbook so that Nikhil, Adi, and Mihir can independently test optimizations without needing to wait for me or understand all the measurement details.

Specific commitment: By next checkpoint, there will be a documented, reproducible process for:

  • Running a baseline benchmark
  • Measuring TTFT (cold/warm), TPS, GPU utilization
  • Comparing before/after results
  • Documenting findings in a standard format

This turns measurement knowledge into team infrastructure.

Evidence I will collect during the next sprint

  • Baseline TTFT (cold/warm) with test methodology documented
  • Baseline TPS with standard prompt and test harness
  • GPU utilization breakdown (memory bandwidth vs compute vs I/O)
  • Comparison table: ollama vs llama.cpp performance on our hardware
  • Model warming implementation with before/after TTFT measurement
  • Speculative decoding feasibility assessment or working prototype
  • Quantization comparison table (Q4, Q5, Q6 levels and their impact)
  • GPU driver + CUDA environment working and documented
  • Shared benchmarking harness with runbook
  • Performance roadmap updated with measured baseline and confirmed priorities
  • At least one optimization fully implemented and measured (not only researched)
  • Failure test: stop a service/worker and document behavior

TECHNICAL EVALUATION SCORING — SUMMARY TABLE

Category Self Rank Notes
T01: Development Environment 0.90 Strong independent Linux/GPU work; diagnostic troubleshooting.
T02: VS Code 0.89 Comfortable with remote dev; evaluated tool overhead.
T03: Git/GitHub 0.92 Excellent issue-driven workflow and documentation discipline.
T04: Portfolio/Pages 0.88 Works well; could improve reproducibility.
T05: Tools & Equipment Hacks 0.91 Adapted techniques to hardware constraints; research-backed.
T06: Portfolio/Blogging 0.87 Structured documentation; advanced customization not focus.
T07: Theme/Style/Layout 0.87 Clear documentation structure; not primary sprint focus.
T08: JavaScript/Coding 0.86 Inference engineering is Python/systems; foundation exists.
T09: AI Orchestration 0.92 Strong architecture understanding; control plane, layers, tradeoffs.
T10: Tech/Cyber Growth 0.91 Expanded from single-task to systems thinking and optimization.
T11: Learning Through Mistakes 0.93 Consistently uses errors to improve mental model; excellent.
T12: Tech/Cyber Talk 0.90 Clear communication at multiple levels; documentation excellent.
Tools & Dev Environment Avg 0.90
Development & Creation Avg 0.89
Technical Awareness & Communication Avg 0.91
OVERALL TECHNICAL AVERAGE 0.90
Category Self Rank Notes
P01: Attendance/Tardy 0.87 Maintained progress; some timing inconsistencies.
P02: Work Habits 0.91 Investigates thoroughly; goes beyond minimums.
P03: Integrity 0.93 Clear about working/proposed/uncertain; honest limitations.
P04: Communication 0.90 Detailed GitHub issues; could sharpen task specificity.
P05: Help Seeking 0.89 Asks for feedback on approach; uses feedback well.
P06: Mentoring/Advocacy 0.90 Documentation helps team; could pair more intentionally.
P07: Timeliness 0.88 Completes research; could improve closure speed.
P08: Persistence 0.93 Excellent; continues through difficulty. Strong skill.
P09: Organization 0.90 Issues, tables, phases, metrics; clear structure.
P10: Engagement 0.92 Highly engaged; research questions, tradeoffs, strategies.
Core Behaviors Avg 0.90
Collaboration Avg 0.90
Professional Skills Avg 0.91
OVERALL PROFESSIONAL AVERAGE 0.90

Overall Ground Zero Average

Technical Average: 0.90
Professional Average: 0.90
Combined Ground Zero Average: 0.90

14. Questions I Want the Next Sprint to Answer

  1. Baseline: What is the actual TTFT and TPS on cold/warm starts with our hardware and model?
  2. Bottleneck: Is the limiting factor memory bandwidth, compute, or I/O?
  3. Warming: How much does aggressive model warming reduce cold-start TTFT?
  4. Runtime: Does ollama or llama.cpp give better performance for our workload?
  5. Quantization: Which GGUF quantization level balances VRAM, speed, and accuracy?
  6. Speculative Decoding: Is implementing speculative decoding worth the complexity for our hardware?
  7. Multi-GPU: Would distributing inference across multiple GPUs actually improve throughput?
  8. Observability: What metrics should we collect and expose for monitoring?
  9. Reliability: How stable is the inference engine under sustained load?
  10. Cost: What is the actual electricity/operating cost per inference?

15. Final Ground Zero Reflection

Ground Zero was successful for me because I moved from "can I run a model" to "which engineering choices will make this model useful for students?"

The biggest lesson is that inference is not a single problem; it is a system of interconnected optimization tradeoffs:

  • Quantization affects VRAM, latency, and quality.
  • Model warming affects cold-start experience and VRAM cost.
  • Runtime choice affects TTFT, TPS, and development complexity.
  • Speculative decoding affects throughput but adds complexity.
  • GPU selection affects available optimization techniques.

The project is now at a better starting point. The goal is no longer "just run Qwen3.8 on the GTX 1070." The goal is to build a measured, optimized inference platform where:

  • Every optimization is backed by before/after measurement
  • Every tradeoff is documented
  • Every choice is reproducible and understandable
  • Performance targets are ambitious but real (60+ TPS, <2s TTFT)
  • Students get free, high-quality AI without expensive subscriptions

My next step is to stop planning optimizations and start measuring baselines. If I can establish reproducible benchmarks and run the first optimization (model warming) to completion, we will have a real foundation for the optimization roadmap.

Ground Zero Close-Out

Current status: ✅ Ready to move from environment setup into measurement and optimization

Primary next milestone: Baseline inference performance measured and documented for the entire team.

What I want my next checkpoint to prove:

I can turn the inference optimization roadmap into working implementations, measure results objectively, and present clear before/after evidence that each optimization actually works.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions