Yash - Ground 0

[illuminati1618]

Ground Zero Check — CSH / OCS Intelligence Hardware & Systems Architecture

Sprint focus: Move from onboarding into project development by proving I can independently diagnose and rebuild physical infrastructure, reason through systems architecture decisions (including ones that get overturned), and run a trio's coordination as scrum master.

Project direction: Bring Rig 2 online as a functioning second GPU rig for the OCS Intelligence LLM system, while owning the systems-architecture thinking that shaped the project's hardware direction and organizing the trio (myself, Nikhil, and our third teammate) so both halves of the group stay consistent.

---

Executive Summary

My biggest change during Ground Zero was moving from thinking about my role as "get the hardware working" to understanding it as two connected problems: physical infrastructure recovery and systems architecture design, both of which needed to hold up under real constraints instead of staying theoretical.

Rig 2 did not arrive usable. GPU holders were physically broken and power was disconnected — not a driver problem, not a config problem, a literal hardware problem. That meant the actual first step wasn't installing CUDA or benchmarking anything; it was taking the rig apart and rebuilding it component by component before any software work could even begin.

At the same time, I was doing the architecture thinking that shaped where this project was headed before it became what it is now. My original design was a three-rig split — one rig dedicated to LLM serving, one built around fast plus persistent storage, and one for ML work with its own persistent data storage — before the team pivoted to the redundant dual-LLM-rig system that's now documented in the group's main architecture. That earlier design isn't wasted work; it's evidence of the reasoning that led to the current direction, and I can explain why we moved away from it.

flowchart TD
    subgraph Rig2["Rig 2 Rebuild"]
        A["Rig 2 arrives"] --> B["GPU holders broken, power disconnected"]
        B --> C["Full teardown"]
        C --> D["Rebuild component by component"]
        D --> E["Still diagnosing remaining issues"]
        E --> F["Target: functional second GPU rig"]
    end

    subgraph Arch["Architecture Evolution"]
        G["Original 3-rig design"] --> H["LLM rig"]
        G --> I["Fast plus persistent storage rig"]
        G --> J["ML plus persistent data rig"]
        G -. pivot .-> K["Redundant dual-LLM-rig system, current"]
    end

    style D fill:#fff3cd,stroke:#e0a800,stroke-width:2px
    style F fill:#fff3cd,stroke:#e0a800,stroke-width:2px
    style K fill:#d4edda,stroke:#28a745,stroke-width:2px

Loading

---

1. Ground Zero Readiness Check

Overall Status

Ground Zero Area	Status	Evidence / Current Position
1. Understand the Challenge	✅ Ready	I understand that Ground Zero here meant proving I can diagnose and fix physical infrastructure, not just follow a rebuild checklist. I went in expecting a software problem and found a hardware problem instead.
2. My Development Environment	✅ Ready	I can work hands-on with rig hardware (teardown, GPU mounting, power) and hold the system-level design context needed to know why the rig needs to look a certain way.
3. My GitHub Workflow	✅ Ready	I set up and organize the team's GitHub Project board to track our workstreams.
4. Forming the Team	✅ Ready	As scrum master of our trio, I've handled stand-ups, task assignment/tracking, and keeping my hardware track in sync with Nikhil's inference-optimization track. Coordination between the two halves has been smooth.
5. Beginning the Project	🟡 Mostly Ready	The architecture direction is clear and I can defend it, but Rig 2 itself is still mid-rebuild, not yet a proven, working second node.

Ground Zero Transition

flowchart LR
    A[Assume Rig 2 is a software setup task] --> B[Discover physical damage]
    B --> C[Full teardown]
    C --> D[Diagnose each broken component]
    D --> E[Rebuild]
    E --> F[Still diagnosing]
    F --> G[Functional Rig 2]

    style A fill:#f8d7da,stroke:#dc3545,stroke-width:2px
    style D stroke-width:3px
    style F stroke-width:3px
    style G fill:#d4edda,stroke:#28a745,stroke-width:2px

Loading

The important shift for me is realizing that "onboarding" on the hardware track didn't mean installing drivers first — it meant confirming the physical machine could even hold a GPU correctly before any of that mattered.

---

2. Understand the Challenge

What I Think the Ground Zero Challenge Actually Was

The challenge wasn't to get Rig 2 running by following a setup guide, because no guide covers "the GPU holders are broken and the power is disconnected." The real challenge was figuring out how to diagnose and rebuild a physical system with no clear starting checklist.

My process became:

Open it up → find what's actually broken → rebuild the broken piece → confirm it holds → move to the next piece

Example: Rig 2 Physical Rebuild

When I started working on Rig 2, the GPU holders were physically broken and the power connections were disconnected — this wasn't a case of bad drivers or misconfiguration, it was a hardware assembly problem. I took the rig apart entirely and have been rebuilding it piece by piece rather than trying to patch around the damage. That's still in progress; I'm partially through the rebuild and still diagnosing remaining issues before I can call it a working second rig.

That experience reframed the problem for me. "Bring up Rig 2" sounds like a software task until you actually open the case. A big part of Ground Zero for me was learning to check the physical layer before assuming the problem lives in software.

Example: Architecture That Got Overturned

Before the team settled on the current redundant dual-LLM-rig system, I was the architecture engineer behind an earlier design: three separate rigs, each with a distinct role — one dedicated to LLM inference, one built around fast plus persistent storage, and one for ML workloads with its own persistent data storage. That design held together on paper, but the team moved to the simpler redundant two-rig LLM system instead.

I'm including that earlier design here on purpose. It's real evidence that I can reason through a systems architecture problem, not just accept whatever direction was handed to me — and understanding why we moved away from it is part of my own technical growth this sprint.

---

3. My Development Environment

Current Working Environment

Area	What I Can Do Independently	Ground Zero Evidence
Hardware Assembly / Rebuild	Disassemble and reassemble a GPU rig, diagnose broken physical components (GPU holders, power connections)	Rig 2 teardown and in-progress rebuild
Systems Architecture	Design a multi-node hardware split around distinct roles (compute vs. storage vs. ML), and revise that design under team feedback	Original 3-rig architecture, superseded by the current redundant LLM system
GitHub Project Management	Set up and maintain a GitHub Project board to organize team workstreams	https://github.com/orgs/Open-Coding-Society/projects/5/
Team Coordination	Run stand-ups, assign and track tasks, keep parallel workstreams (hardware vs. inference optimization) in sync	Scrum master role across the trio

My Environment Strategy

My strategy on the hardware side has been to treat Rig 2 as a full rebuild rather than a repair-in-place. Since the damage was structural (broken GPU holders, disconnected power), patching individual symptoms risked missing something else that was also wrong. Taking the whole rig apart let me verify each component instead of assuming the rest of the rig was fine.

flowchart TD
    A[Rig 2, damaged on arrival] --> B[Full teardown]
    B --> C[Inspect every component individually]
    C --> D[Rebuild GPU mounting]
    D --> E[Rebuild power connections]
    E --> F[Continue diagnosing remaining issues]
    F --> G[Bring rig to parity with Rig 1]

    style A fill:#f8d7da,stroke:#dc3545,stroke-width:2px
    style G fill:#d4edda,stroke:#28a745,stroke-width:2px

Loading

---

4. My GitHub Workflow

I set up and organize the team's GitHub Project board, which is where our trio's hardware and inference-optimization workstreams are tracked side by side:

• Team project board: https://github.com/orgs/Open-Coding-Society/projects/5/

As scrum master, my workflow is less about individual commits and more about keeping the board an accurate reflection of where both halves of the team actually are:

flowchart LR
    A[Stand-up] --> B[Confirm task status per track]
    B --> C[Update project board]
    C --> D[Flag blockers]
    D --> E[Sync hardware track with inference track]
    E --> F[Next stand-up]
    F --> A

Loading

Why This Is Better Than My Starting Workflow

Earlier, task tracking was more ad hoc — work happened but wasn't always visible to the rest of the team. Organizing a shared project board means Nikhil's inference-optimization work and my hardware/architecture work are both visible in one place, so neither track has to guess what the other is waiting on.

---

5. Forming and Operating as a Team

What Is Working

As scrum master of our trio, I've handled all three core pieces of team organization:

• Stand-ups / check-ins to keep the trio aligned
• Task assignment and tracking through the GitHub Project board
• Syncing my hardware track with Nikhil's inference-optimization track so both halves of the group stay consistent

Coordination between the hardware track and the inference-optimization track has been smooth — there hasn't been a recurring blocking dependency between the two, which has let both of us make progress in parallel rather than waiting on each other.

Workstream	Who Owns It
Hardware (Rig 2 rebuild)	Me
Systems Architecture	Me (original 3-rig design → current redundant LLM system)
Team Coordination / Scrum	Me
Inference Optimization	Nikhil (+ team)

Scrum Master Loop

flowchart LR
    A[Stand-up] --> B[Check hardware track status]
    A --> C[Check inference track status]
    B --> D[Update project board]
    C --> D
    D --> E{Cross-track blocker?}
    E -->|No| F[Both tracks proceed independently]
    E -->|Yes| G[Flag and resolve dependency]
    F --> A
    G --> A

    style E fill:#fff3cd,stroke:#e0a800,stroke-width:2px
    style G fill:#f8d7da,stroke:#dc3545,stroke-width:2px

Loading

What Still Needs Improvement

The main open item isn't team process — it's that Rig 2 itself isn't finished. The next sprint needs to convert "partially rebuilt, still diagnosing" into a rig that's actually verified working, ideally with the same kind of benchmark evidence the inference track is already collecting on Rig 1.

---

6. Beginning the Project — Rig 2 & Systems Architecture

Project Problem

The question I'm responsible for isn't "can a GTX 1070 run an LLM" — that's already been answered on Rig 1. My question is:

Can a second, physically damaged rig be rebuilt from the ground up to match Rig 1's capability, giving the OCS Intelligence system real redundancy instead of a single point of failure?

Hardware Direction

Rig	Status	Role
Rig 1	Functional	Production inference serving
Rig 2	Partially rebuilt, still diagnosing	Redundancy / development capacity once online

Architecture Evolution

Design	Status	Reasoning
Original 3-rig split (LLM rig / fast+persistent storage rig / ML+persistent data rig)	Superseded	Clean separation of concerns by role, but added complexity the team didn't need yet
Current redundant dual-LLM-rig system	Active direction	Simpler, gives the project actual failover instead of three specialized single points of failure

flowchart LR
    subgraph Before["Before — 3 Specialized Rigs"]
        direction TB
        R1["Rig A: LLM serving"]
        R2["Rig B: Fast + persistent storage"]
        R3["Rig C: ML + persistent data"]
    end

    subgraph After["After — 2 Redundant Rigs"]
        direction TB
        N1["Rig 1: LLM serving"]
        N2["Rig 2: LLM serving, redundant"]
    end

    Before -->|pivot decision| After

    style R1 fill:#e2e3e5,stroke:#6c757d
    style R2 fill:#e2e3e5,stroke:#6c757d
    style R3 fill:#e2e3e5,stroke:#6c757d
    style N1 fill:#d4edda,stroke:#28a745,stroke-width:2px
    style N2 fill:#fff3cd,stroke:#e0a800,stroke-width:2px

Loading

The single points of failure in the original design (one rig per role, nothing backing it up) are why the team moved to two rigs that can cover for each other instead of three rigs that each cover a different function.

---

7. Technical Evaluation

Technical Score Summary

Category	Self Rank
Hardware / Hands-On Rebuild	4 / 5
Systems Architecture / Design	4 / 5
Overall Technical Average	4 / 5

Reflection — Hardware / Hands-On Rebuild

I'm comfortable doing the hands-on teardown-and-rebuild work independently — diagnosing that the GPU holders and power connections were physically broken didn't require someone walking me through it. I'm ranking this a 4 rather than a 5 because some of the remaining diagnosis on Rig 2 is still new territory; I haven't hit every possible failure mode yet, and I want to see the rig fully verified working before I'd call this a 5.

Reflection — Systems Architecture / Design

I can explain and defend both the original 3-rig design and the reasoning behind the pivot to the current redundant system. I didn't just accept the change — I understand why the simpler two-rig redundant approach won out over three specialized rigs. That's a 4 because the design has already proven it can survive real feedback and change; a 5 would mean I've also proven the current architecture works at scale, which hasn't happened yet.

---

8. Professional Evaluation

Professional Score Summary

Category	Self Rank
Scrum Master / Team Organization	4 / 5

Reflection — Scrum Master / Team Organization

The team runs smoothly, and I think that's a direct result of owning all three pieces of coordination — stand-ups, task tracking, and keeping the hardware and inference tracks in sync — rather than just one of them. Coordination between my track and Nikhil's hasn't produced real friction so far. I'm ranking this a 4 instead of a 5 because the real test of a scrum process is how it holds up once something does block across tracks (e.g., if inference optimization work ends up needing Rig 2 online before it's ready) — that hasn't happened yet, so I want to see the process survive a real cross-track blocker before calling it a 5.

---

9. Next Sprint Commitment

One professional behavior I will improve

Get ahead of potential cross-track blockers before they happen — right now coordination has been smooth because nothing has forced Nikhil's track to depend on Rig 2 being ready. I want to proactively flag that dependency risk instead of discovering it reactively.

One technical skill I will improve

Finish diagnosing and rebuilding Rig 2 to full parity with Rig 1, including whatever driver/CUDA verification comes after the physical rebuild is actually done.

One way I will contribute more effectively to my team

Once Rig 2 is verified working, document its rebuild and specs the same way Rig 1's benchmarking is documented, so the team has a matched pair of rigs instead of one proven rig and one unknown.

Evidence I will collect during the next sprint

• Rig 2 fully reassembled with GPU holders and power confirmed stable
• Rig 2 boots and OS/drivers recognize all GPUs
• Photo/log evidence of the rebuild process
• Rig 2 added to the GitHub Project board as a tracked, verified component
• Basic benchmark on Rig 2 to confirm parity with Rig 1
• Updated architecture diagram reflecting Rig 2 coming fully online

---

10. Final Ground Zero Reflection

Ground Zero was useful for me because it forced me to separate two things I'd been treating as one: the design of the system and the physical reality of the hardware running it. I came in with a real architecture behind me — the original 3-rig split — and had to learn that a good design on paper still has to survive contact with a rig that arrived with broken GPU holders and disconnected power.

The rebuild isn't finished yet, and I'm not going to pretend it is. What I do have is a clear, honest picture of where Rig 2 stands, a defensible explanation for why the project's architecture changed under me, and a team that's staying coordinated because I'm actively running that coordination as scrum master.

Ground Zero Close-Out

Current status: 🟡 Ready to move from onboarding into project development, with Rig 2's physical rebuild as the primary open item

Primary next milestone: Rig 2 fully rebuilt, verified, and benchmarked to parity with Rig 1.

What I want my next checkpoint to prove:

I can take a physically damaged piece of infrastructure and bring it to full working parity with the rest of the system, while continuing to run team coordination that keeps both workstreams moving.