Knowledge Management

A personal knowledge management repository for organizing notes, references, and insights. Includes an automated pipeline for converting PDFs and web pages to markdown, organized by subject area, with sync to Obsidian.

Structure

knowledge-management/
├── notes/                        # Atomic notes, fleeting thoughts, and evergreen content
├── references/
│   └── papers/                   # Converted markdown output, organized by subject
│       ├── transformers/
│       │   ├── 1706.03762.md
│       │   └── 1706.03762_images/
│       └── cuda/
├── resources/
│   └── sources/                  # Input sources, organized by subject
│       ├── transformers/
│       │   └── urls.txt
│       └── cuda/
│           └── urls.txt
├── scripts/
│   ├── convert_pdfs.py           # Conversion script (run locally in .venv)
│   └── sync_to_vault.py          # Local Obsidian vault sync script
├── projects/                     # Project-specific knowledge and documentation
├── templates/                    # Reusable document and note templates
├── sync_config.json              # Subject → Obsidian vault path mapping
├── requirements.txt              # Python dependencies (pymupdf4llm, markitdown)
└── README.md

PDF & Web Page Conversion Pipeline

The pipeline converts PDFs and web pages to markdown, extracts images from PDFs, and syncs the output to your Obsidian vault with proper frontmatter for the wiki schema.

How it works

1. Add source URLs to resources/sources/<subject>/urls.txt
2. Convert locally: .venv/bin/python scripts/convert_pdfs.py downloads PDFs and converts every source to markdown (with images). It is incremental — a source whose markdown already exists is skipped, so re-running is safe and cheap.
3. Review the diff and commit
4. Sync to Obsidian with .venv/bin/python scripts/sync_to_vault.py

The sync script renames files to the wiki schema (<source_type>-<slug>.md), prepends YAML frontmatter, and copies extracted images alongside the markdown.

urls.txt format

Each line follows the format: url | title | source_type [| author]

The author field is optional but recommended for sites whose URLs have generic path stems (e.g. YouTube, where every video shares /watch). When provided, it is appended to the title slug to form the output filename, e.g. llm-inference-lecture-roofline-analysis-for-gpu-faradawn-yang.md.

https://arxiv.org/pdf/1706.03762.pdf | Attention Is All You Need | paper
https://docs.nvidia.com/cuda/cuda-programming-guide/index.html | CUDA Programming Guide | doc
https://www.youtube.com/watch?v=7EJjdDLK4cg | LLM Inference Lecture: Roofline Analysis for GPU | video | Faradawn Yang

Valid source types: paper, blog, video, course, code, thread, pdf, doc

Blank lines and lines starting with # are ignored.

Adding a source to an existing subject

1. Add the URL to resources/sources/<subject>/urls.txt
2. Commit and push
3. Convert:

   .venv/bin/python scripts/convert_pdfs.py

   For a reviewable diff on a larger batch, do it on a branch first (git checkout -b corpus/<name>) and merge once the output looks right.
4. Check the new files — a suspiciously small .md usually means the fetch hit a login wall or a JS-only page. Delete those rather than syncing a blank Raw file; the urls.txt line stays, so a later run retries.
5. Commit, then run .venv/bin/python scripts/sync_to_vault.py

Note on YouTube URLs: YouTube and similar sites use a generic path stem (/watch) for every video, which would cause filename collisions. The script resolves this by building a human-readable slug from the title and author fields: <title-7-words>-<author-3-words>.md. The author field in urls.txt is what enables the author part of the slug — add it whenever you submit a YouTube or similarly generic URL. If no title or author is available, the script falls back to the URL's unique query parameter (e.g. ?v=VIDEO_ID).

Adding a new subject area

1. Create the subject folder with a urls.txt:

   mkdir -p resources/sources/<new-subject>

   Add URLs to resources/sources/<new-subject>/urls.txt

2. Add the Obsidian vault mapping in sync_config.json:

   {
     "subjects": {
       "<new-subject>": "/path/to/obsidian/vault/Raw"
     }
   }

3. (Optional) Register a NotebookLM notebook for the subject so new markdown is synced to NotebookLM via the nlm CLI. Add the notebook ID under notebooklm in sync_config.json:

   {
     "notebooklm": {
       "<new-subject>": "<notebook-id>"
     }
   }

   If omitted, only the NotebookLM push is skipped for this subject — the Obsidian vault sync (sync_to_vault.py) still runs normally.

4. Create the Obsidian vault for the subject area with the required layer folders:

   mkdir -p "/path/to/obsidian/vault/<new-subject>"/{Raw,Wiki,"Learning Path"}
   cp "/path/to/obsidian/vault/<existing-subject>/CLAUDE.md" "/path/to/obsidian/vault/<new-subject>/CLAUDE.md"

   Then adapt the cloned CLAUDE.md to the new domain — strip every term, example, and Learning Path stage scope that belongs to the sibling subject, and substitute domain-native terms. A schema clone is a starting point, not a drop-in.

5. Run the vault bootstrap workflow (defined in the vault's own CLAUDE.md, section "Bootstrap workflow"). This is a mandatory one-time step for every new subject-area vault. The bootstrap creates Wiki/index.md, Wiki/log.md, Wiki/overview.md, plus stub entity/concept pages for the domain's core terms, plus empty Learning Path/ stage files. Without bootstrap, the first ingest has nothing to link into — summaries end up as orphans and the wiki graph-connectivity invariant is broken.

6. Commit, push, and follow steps 3-5 from the "Adding a source to an existing subject" section above.

Resuming a session on any vault

When starting a new session — on any subject-area vault in this knowledge base — the LLM has no memory of prior work. Before giving any new instructions, ask it to orient itself by reading the load-bearing state files so it can pick up where the last session left off without guessing.

The three checks, in order:

1. Read Wiki/log.md for the vault you're working in — the append-only log tells you what's been ingested, what stubs were created, what's deferred, and what was touched last. This is the single most load-bearing file for session continuity.
2. Check Raw/ for unprocessed files — anything newer than the latest log entry is pending ingest.
3. Glance at git status and recent commits — catches anything changed outside the log (config updates, CLAUDE.md edits, new subject areas, URL additions).

Suggested resume prompt (vault-agnostic — just substitute the vault path):

Review Wiki/log.md in the <vault-name> vault, check Raw/ for unprocessed files, glance at git status / recent commits, and tell me what state we're in before we start. Don't edit anything yet.

If you're working across multiple vaults in one session (e.g., bouncing between the Transformer and CUDA vaults), ask for the orientation check on each vault explicitly — the LLM will only check the vault you name.

The vault's own CLAUDE.md contains a longer-form version of this resume prompt tailored to that vault. Use the short form for daily interactive sessions; reach for the long form only when the vault is unfamiliar or it's been weeks since the last session.

Converters used

• PDFs: pymupdf4llm — lightweight, extracts images inline at their original position
• Web pages: markitdown (Microsoft) — converts HTML URLs directly to markdown

Local conversion environment

Conversion runs locally in a virtualenv (.venv/, git-ignored):

uv venv --python 3.12
uv pip install -r requirements.txt

Then always invoke the scripts through it: .venv/bin/python scripts/convert_pdfs.py.

The youtube-transcription extra is load-bearing. requirements.txt pins markitdown[youtube-transcription], not bare markitdown. Without the extra, markitdown returns only a video's title and description — a 13,000-word lecture becomes a 200-word shell, and the failure is silent. If video conversions come back at a few hundred words, that extra is missing.

(This replaced a Warp cloud agent that ran the same script and opened a PR. The only thing lost is the PR review gate; use a branch for large batches instead.)

Working practices for agents

Adopted 2026-08-29 after a corpus-expansion session where checking beat assuming, every time. Any agent working in this repo or its vaults — Claude, Codex, or otherwise — should follow these. They are cheap; the mistakes they prevent are not.

1. Never classify or route a source by its title or by keywords. A first-match regex classifier sent Unsloth to inference, nanoGPT to fine-tuning, and backprop to transformers, because it cannot tell "quantization for serving" from "quantization during fine-tuning". Read the captured content. When a page serves a useless <title> (Medium, Google Colab), recover the real title from the URL slug rather than discarding the record — one such record was Karpathy's "Yes you should understand backprop".

2. Before moving or deleting a source, grep the vault for inbound references. nanochat/gpt.py reads as a "build a ChatGPT" repo and was routed out of the inference vault on that basis; it is in fact cited by Wiki/entities/KV Cache.md as the production reference building the cache on the FlashAttention flash_attn_with_kvcache kernel. Titles mislead; citations don't.

3. Ask what would undo the change. Deleting a Raw/ file is not durable by itself: scripts/sync_to_vault.py recreates any destination that no longer exists, so the daily launchd sync restores it unless the urls.txt line and the converted references/papers/ files move too. Trace the pipeline to its end before calling a cleanup done, and dry-run the sync to prove it.

4. Re-verify state established earlier in the session. Other sessions edit this repo and these vaults concurrently; during that session a commit and three vault files changed underneath the work in progress. Re-read git log / git status / mtimes instead of trusting an earlier reading.

5. Distrust your own aggregate numbers. A URL-overlap count was reported wrong because the normalizer stripped query strings, collapsing every YouTube link to youtube.com/watch. When a number is surprising, re-derive it a second way before reporting it — and correct it plainly when it was wrong.

6. Back up before irreversible work; verify after. The vaults are not git-tracked. cp -R the vault first, then confirm with python3 scripts/test_okf_bundle.py "<vault>" that the bundle is still conformant and no orphan links remain.

See projects/llm-corpus-expansion/routing.md for these applied to a concrete decision, and each vault's CLAUDE.md for the step-by-step source relocation/removal workflow.

Getting Started

Clone the repository and install dependencies:

git clone https://github.com/aadehamid/knowledge-management.git
pip install -r requirements.txt