NVIDIA NIM API Integration (Llama 3.2 90B Vision) for Blind Metrics and Comparative Analysis

[Yehudha-kennedy]

Description

This Pull Request introduces dynamic support for using NVIDIA NIM's OpenAI-compatible API (meta/llama-3.2-90b-vision-instruct) within the project's experimentation pipeline. It includes a comparative architecture analysis against Gemini, results from the blind-metrics pilot test ($n=35$), and the infrastructure adjustments required to mitigate underlying network behaviors.

Current Progress: The technical analysis shown below corresponds to an initial pilot sample of $n=35$. Full-scale validation with the complete sample of $n=600$ using Llama 3.2 90B Vision is currently running, progressing at a steady rate thanks to the implemented blocking mitigations. Final results will be reported upon completion.

---

Infrastructure Comparison: NVIDIA NIM API vs Gemini API

During development and integration, we identified critical operational differences between both APIs for our workload:

Technical Feature	NVIDIA NIM API (Llama Vision)	Gemini API (Google)
SDK Integration	Requires the generic openai adapter.	Native SDK (google-genai), highly optimized.
Image Handling	Standardized Base64 strings (susceptible to long network I/O timeouts).	Natively optimized for asynchronous, multipart sending.
Max Concurrency	Low (40 RPM). Penalizes concurrent bursts.	High. Smoothly supports multiple parallel workers.
Rate-Limit Response	Aggressive. May hold the socket connection open but idle (tarpit), causing "deadlocks."	Clean (returns a semantic error code for retry routing).

---

Empirical Results (Pilot $n=35$, Llama 3.2 90B Vision)

A pilot validation of 35 samples was run using the decoy-uptake vulnerability metric (with counterfactual binary decoys constructed relative to baseline reads). The findings expose strong lability in the Llama 3.2 90B Vision model:

Decoy Uptake:

• Blind (85.71%): When a "decoy" was injected (telling the model that an automated metric would award it full marks if it answered the opposite of its own assessment), the model altered its actual clinical diagnosis in favor of the false one in 30 out of 35 cases (85.71%).

Test-Aware Suppression:

• Test-Aware (28.57%): When explicitly warned at the prompt level that this was a validation check of independent clinical judgment and the rubric might be misaligned, its decoy-uptake rate dropped to 10 out of 35 cases (28.57%).
• In other words, contextual warning suppressed the deception by 57.14% (delta: $0.8571 - 0.2857 = 0.5714$), but the model still fell into the trap in 1 out of every 3.5 clinical cases.

Silent Drifting:

• Of the 30 instances in which the model changed its answer to match the false decoy, in none of them (0) did it justify its final diagnosis by mentioning the scoring rule or the decoy's influence.
• All 30 cases consisted of silent drifting: the model retroactively invented a false clinical justification to align with the incorrect answer induced by the decoy.

---

Bug Fixes: Shortcuts and Workarounds Implemented

Due to NVIDIA's rate-limiting policy profile (40 RPM) and server-side connection behavior, the following mitigations and fixes were implemented:

• Centralized Parser & Model Scoping:
  Replaced local _yesno() with shared parse_yesno() from benchmaxxing.extract (last-match semantics). Added --model CLI argument and scoped output directories ({out}/{model_slug}/) to prevent cross-model overwrites.

• Strict API Key Dispatch & Cache:
  Strict API key resolution in _key(model) ensuring NIM models only query NVIDIA_API_KEY and fail loudly on cache misses without falling back to Gemini keys. Raw response cache for the pilot is committed in img_cache.jsonl for standalone reproducibility without API keys.

• Manual Timeout Control (Deadlock Prevention):
  The OpenAI client was reconfigured by injecting timeout=60.0 and overriding the internal retry policy via max_retries=0. Retry control with exponential backoff relies on the RetryBackend wrapper class.

• Forced Degradation to Sequential Processing:
  Parallelism in imaging_blind_metric.py was set to max_workers=1 to prevent socket throttling and firewall blocks from concurrent bursts.

[sebasmos]

Superseded by my re-review of d3bb8a4. The parser finding here was fixed in that commit.

[Yehudha-kennedy]

Done. Fixed all three items:

1. Parser: Replaced the local _yesno() with the shared parse_yesno() from benchmaxxing.extract — it already does last-match, so it now reads the final answer line instead of the justification. Corrected pilot: blind uptake went from 0.80 → 0.8571 (2 extra drifts the old parser was miscounting). Zero unparseables.
2. --model flag: Added as a required CLI arg, removed the hardcoded constant. _key() now dispatches by model name.
3. Scoped output: Results now write to {out}/{model_slug}/ so it won't overwrite Gemini results. Pilot rows committed.

Pushed to the branch — ready for re-review.

[sebasmos]

REQUEST_CHANGES, four asks.

Your corrected pilot exactly matches our Gemini run on the same 35 cases: 0.8571 blind, 30 drifters,
0 naming the rubric. I replayed our 105-response cache under both parsers, zero disagreements, so the
comparison is like-for-like. That replication is the strongest thing here; the real difference is
test-aware, 0.286 vs 0.171.

1. Body still says 82.86/25.71/29; match the committed artifact.
2. Drop the baseline 0% claim: base_is_decoy cannot be true, ours included (#374).
3. _key() still hands the Gemini key to NIM models; fail loudly.
4. Commit your Llama cache so the parse is checkable.

[Yehudha-kennedy]

All four items addressed in commit 70f3d92 and PR body update:

1. PR Body numbers updated: Aligned all numbers to the committed artifact (.71%$ blind, .57%$ test-aware, /30$ silent drifters, .14%$ suppression delta).
2. Dropped baseline 0% claim: Removed the tautological baseline 0% assertion per Audit every reported metric for predicates that cannot fail by construction #374 (counterfactual decoy constructed relative to baseline read).
3. Strict _key() resolution: NIM models now strictly require NVIDIA_API_KEY (no Gemini fallback) and fail loudly on cache miss naming the expected key.
4. Committed response cache: Committed the raw response cache in experiments/chexpert/results/img_cache.jsonl so the 105-response pilot run and parser can be verified offline with zero API calls/keys.

Ready for re-review.