MultiNet 2.0 Preview: Goal Progress Decays with Task Horizon for Frontier VLMs in Interactive 2D Environments
MultiNet is a collaborative initiative with contributions from leading research teams at institutions like:
Interested in evaluating your agent on long horizon tasks, or collaborating on research? Let's work together
Claude Opus 4.8, Kimi K2.6 and Qwen 3.6 27B models failing on 2D mazes.
- 🌀 2026-08-26: A Preview of our Multi-Domain Agentic Benchmark - We evaluate 3 frontier VLMs on 50 2D mazes to understand how, where, and why they break in an environment that requires exploration, planning, long-horizon action taking, and causal reasoning. Read our technical report here.
- 🎓 2026-04-03: Paper accepted at CVPR 2026! Our work has been accepted at the MMFM Workshop at CVPR 2026! Read our paper here.
- 🌟 2025-10-13: Multinet v1.0 - We release our most comprehensive benchmark yet - evaluating a SoTA VLM, VLA, and generalist model on a wide variety of multimodal understanding and action datasets. Read more here
- 🏅 2025-06-10: Paper accepted at ICML 2025! Our paper detailing the Open-Source contributions of Multinet that benefit the AI community has been accepted at the CodeML Workshop at ICML 2025! Read our paper here.
- 🏆 2025-05-22: Multinet v0.2 - We systematically profile state-of-the-art VLAs and VLMs to understand how they perform in procedurally generated OOD game environments! Read more about our release here
- 🎉 2024-11-08: We release the first version of MultiNet where we profiled SoTA VLMs and VLAs on real-world robotics tasks - Multinet v0.1! Check our release page for more details.
- 🚀 2024-03-22: Introducing Multinet! A new generalist benchmark to evaluate Vision-Language & Action models. Learn more here
With MultiNet v2.0 we aim to build interactive environments that are proxies for real-world scenarios. However, at the same time we are keen to keep the setup controllable, which will allow us to deterministically vary parameters in our environment in order to make it easier or more difficult for models to succeed in.
The capabilities we aim to benchmark are long-horizon action taking and causal reasoning, which involves various sub-capabilities such as planning, action execution, error recovery, visual object association, and so much more. A simple underlying substrate that brings all these aspects together for an environment and benchmarking task is a maze with mechanisms. Additionally, mazes with mechanisms are projectable into multiple domains: the same maze can be re-rendered in language or 3D or many other domains, quantifying cross-domain generalization. In this release, MultiNet 2.0 Preview: Interactive 2D Mazes, we evaluated 3 frontier VLMs on 50 2D mazes.
- The environment: 8×8 to 14×14 MiniGrid mazes with an action space containing 6 valid actions: turn left, turn right, move forward, pickup, toggle, and done. The agent must navigate corridors, dead ends, distractors and decoys, operate mechanisms in the right order and reach a goal tile.
- A validator and BFS oracle: every maze is confirmed solvable, with checks for mechanism necessity, chain ordering, and distractor safety. The oracle yields the exact optimal action sequence from any reachable state, giving objective difficulty, partial credit, and the ability to label a single move as strictly wrong.
- An evaluation harness: a config-driven episode runner (prompt assembly, strict action parsing, per-episode artifact logging, a progress-stall watchdog, difficulty-relative step caps), model adapters behind one interface, mechanism-aware scoring, and the distributed run infrastructure that executed the evaluation across a fleet of VMs and GPUs.
- An ablation-derived protocol: extensive experiments were run across 540 episodes to finalize the evaluation protocol for the final run on 50 mazes.
We evaluated Claude Opus 4.8 (xhigh thinking), Kimi k2.6 (thinking), and Qwen3.6-27B (thinking) on 50 difficulty-balanced mazes, with an equal 64k output-token budget.
| Claude Opus 4.8 | Kimi k2.6 | Qwen3.6-27B | |
|---|---|---|---|
| Mazes solved (/50) | 4 | 1 | 1 |
| Mean action progress | 0.19 | 0.23 | 0.23 |
6 solves out of 150 episodes. 45 of the 50 mazes were solved by no model at all. These are puzzles a person who has never seen one solves in a few minutes. Try out some of the mazes here and see how you fare!
Progress per maze (columns) per model (rows); stars mark the six solves.
For a deeper dive, read our technical report.
git clone https://github.com/ManifoldRG/MultiNet-v2.0.git
cd MultiNet-v2.0
conda create -n multinet-v2 python=3.10 && conda activate multinet-v2
# (or: python -m venv .venv && source .venv/bin/activate)
pip install -e ".[dev,visual]"Mazes are declarative JSON task specifications. Validate every example spec in the repo and rank them by difficulty:
python -m gridworld.task_validator [PASS] tier3_key_switch_001: optimal=30 steps, mechanisms=4, score=70.61
...
=== Summary: 16/16 tasks beatable ===
To build your own maze, copy a spec from gridworld/tasks/, edit the layout and mechanisms, then validate and render it:
from PIL import Image
from gridworld.task_spec import TaskSpecification
from gridworld.task_validator import compute_difficulty
from gridworld.backends.minigrid_backend import MiniGridBackend
spec = TaskSpecification.from_json("gridworld/tasks/tier3/key_switch_001.json")
report = compute_difficulty(spec)
print(report.is_beatable, report.optimal_steps, report.mechanism_count)
backend = MiniGridBackend()
backend.configure(spec)
backend.reset(seed=0)
Image.fromarray(backend.render()).save("maze.png")compute_difficulty runs the BFS oracle: if your maze is unsolvable, has a decorative mechanism, or has a distractor that can strand the agent, it will tell you.
| Path | Contents |
|---|---|
gridworld/ |
task specification, maze validator, BFS oracle, MiniGrid + MultiGrid backends |
interface/ |
episode runner, prompt assembly, action parsing, model adapters |
prompting_experiments/ |
every prompt template used in the protocol sweep |
scorer/ |
static and runtime scoring, mechanism-aware progress |
demo/ |
the playable maze demo embedded on the website |
scripts/ |
evaluation pipeline entrypoints and run tooling |
deploy/ |
distributed run infrastructure: VM and GPU fleet provisioning, teardown, and cost-safety rails |
Our previous research with MultiNet v1.0 and earlier versions all live in the MultiNet v1.0 repository: evaluations of VLMs, VLAs, and generalist models across a wide variety of domains such as robotics, multimodal understanding, game play, and tool-calling to understand their cross-domain generalization capabilities.
Please cite this work as:
Guruprasad, P., Rivera, S., Lu, H., Jain, A., Ren, H. and Sikka, H. (2026) MultiNet 2.0 Preview: Goal Progress Decays with Task Horizon for Frontier VLMs in Interactive 2D Environments. Available at: https://www.fig.inc/multinet-v2-preview
Or use the BibTeX citation:
@online{multinet_v2_preview_technical_report_2026,
title = {MultiNet 2.0 Preview: Goal Progress Decays with Task Horizon for Frontier VLMs in Interactive 2D Environments},
author = {Pranav Guruprasad and Sean Rivera and Helen Lu and Arushi Jain and Hangliang Ren and Harshvardhan Sikka},
year = {2026},
url = {www.fig.inc/multinet-v2-preview},
note = {MultiNet 2.0 Preview: Interactive 2D Mazes}
}If you build models or agents, or work on benchmarking and evaluation, we would love to hear from you - whether that means getting your model on the MultiNet 2.0 Benchmark, collaborating on the next version of the benchmark, or working with us on what comes after.
🧪 Let's work together · ✉️ Work with us · 💬 Join the Discord
Released under the MIT License.