Benchmark suite: on-device LLM performance across a real-device matrix (tok/s, RAM, TTFT, energy)

[sagar-develop]

Goal

Build a reproducible benchmark suite that measures litertlm-kmp performance on real Android devices and produces public, citable numbers for the README and a results page on urjalabs.in. This closes the README roadmap "benchmark suite" item.

Why this matters

• Credibility: hard, reproducible numbers are the strongest signal for adoption, hiring, and any commercial conversation.
• Baseline for Enhance LiteRT performance: enable GPU (OpenCL/Metal) and NPU (QNN) backends #31: we need a CPU baseline before the GPU/NPU work so we can prove the speedup.
• Differentiation: most local-AI benchmarks target MacBooks / flagship Pixels. Our angle is real mid-range and low-end Android — that's the story nobody else tells.

Device matrix (initial)

Filling in on the laptop session — two devices to start (a phone + a tablet); expand later.

Device	Type	SoC	Total RAM	Effective RAM*	Android	Thermal notes
phone — TBD	phone
tablet — TBD	tablet

* Record both MemTotal and the OEM RAM-expansion-corrected value from HardwareProvider.effectiveRamMb() — the gap between them is part of the story.

Models to cover

For each: record model id, quantization, and on-disk size.

• Gemma E2B (primary, text)
• Gemma E4B (multimodal, where RAM tier allows)
• EmbeddingGemma-300M (ONNX) — RAG embedder
• Cross-encoder reranker (ONNX)
• Whisper (STT) — optional
• Vision path: Gemma E2B/E4B with an image attachment

Metrics to capture

Generation

• TTFT — time to first token (prefill latency)
• Prefill throughput (tokens/sec)
• Decode throughput (tokens/sec, steady-state) ← headline number
• End-to-end latency for a fixed output length (e.g. 256 tokens)

Memory

• Peak PSS/RSS during load + inference
• Model load time (cold vs warm)
• Max context length before OOM / RAM ceiling per device tier

Energy / thermal

• Battery drain per 1000 generated tokens (mAh or %)
• Thermal throttling: decode tok/s over a sustained run (e.g. 5–10 min) to show sustained vs burst

RAG / embeddings

• Embedding throughput (texts/sec and tokens/sec)
• End-to-end RAG query latency (retrieval + rerank + generation)

Multimodal (optional)

• TTFT + decode with an image attached

Methodology (for reproducibility)

• Airplane mode ON; screen on; fixed brightness; app in foreground.
• Battery 50–90%, not charging (avoid charging/thermal skew); record the level.
• Cool-down to ambient between runs; note ambient temp if available.
• Warm-up: 1 discarded run, then N = 5 measured runs → report median + p90.
• Fixed canonical prompt set (define 3: short prompt, long-context prompt, RAG query).
• Determinism: temperature = 0 (or fixed seed) when measuring speed.
• Fixed output token count for decode-throughput runs.
• Record per run: model id + quant, lib/app version (git SHA + release tag), device, Android version, thermal state.

Harness / implementation sketch

• Add a "Benchmark" debug screen in sample-app (or a dedicated :benchmark module) that runs the matrix programmatically:
  • Use the engine's existing token callbacks to separate prefill vs decode timing.
  • Sample memory via Debug.getPss() / ActivityManager.
  • Export results as JSON + CSV to a shareable file.
• Prefer a custom in-app harness over Jetpack Microbenchmark here (model load + permissions + real inference don't fit the microbench harness well).
• Commit raw results under benchmarks/ (JSON/CSV) + a generated markdown table.

Reporting / deliverables

• BENCHMARKS.md (or docs/ page) — methodology + results table, generatable from the JSON.
• README "Performance" section with headline numbers linking to it.
• urjalabs.in results page (e.g. /benchmarks) rendering the same data — feeds the public positioning.
• Move "benchmark suite" in the README roadmap from Future → Done/In-progress.

Acceptance criteria

• Harness runs end-to-end on ≥ 2 devices (phone + tablet).
• Captures TTFT, decode tok/s, prefill tok/s, peak RAM, load time per model.
• Captures embedding throughput + end-to-end RAG latency.
• Results exported as JSON/CSV and a rendered markdown table.
• Methodology documented so a third party can reproduce.
• README + urjalabs results page updated.

Out of scope (for now)

• GPU/NPU backends — tracked in Enhance LiteRT performance: enable GPU (OpenCL/Metal) and NPU (QNN) backends #31; the harness should be reusable to measure those gains later.
• iOS — no device yet.

---

Note: harness should be runnable from the laptop session via Gradle + adb against the two connected devices.