Explain why some routes are less reliably solved

[kctong529]

Goal

Explain which route properties seem connected to easier, harder, or failed LLM route-navigation cases.

Description

Analyze evaluated route results and compare them against route properties.

The goal is not to claim formal statistical significance. Instead, we want a careful qualitative/descriptive explanation of why some routes are less reliably solved.

Possible route properties to inspect:

• reference path length
• number of edges
• number of intermediate nodes
• number of junctions
• average node degree along the route
• maximum node degree along the route
• number of plausible outgoing alternatives
• number of near-shortest alternatives
• difference between shortest and second-best route
• difference between weighted and unweighted shortest path
• whether a detour is required
• whether a swing is required
• whether the locally obvious choice is globally worse
• whether the route is difficult to explain in natural language

Literature / background review

Include a small literature/background review on what makes spatial, graph, or route-navigation tasks difficult.

The review should support the route-property analysis and help explain why some route cases are harder than others.

Useful angles:

• graph size and path length
• branching factor / junction complexity
• shortest-path reasoning
• local vs global route choices
• weighted graph reasoning
• spatial reasoning limits in LLMs

Expected output:

• 3–5 relevant sources
• a short mapping between route properties and expected difficulty
• report-ready text explaining why route difficulty should be analyzed by structure, not only by success/failure

Suggested work

• Select evaluated route cases to inspect.
• Add or collect route-property metadata for those cases.
• Compare model success/failure against route properties.
• Identify examples of easy, hard, and misleading routes.
• Connect examples to the failure-mode taxonomy.
• Avoid overclaiming from small samples.
• Write a short discussion suitable for the final report.

Acceptance criteria

• Route-property fields are defined.
• Selected route cases have relevant properties labeled.
• At least three route examples are discussed.
• The analysis explains why each example was easier or harder.
• The discussion connects route properties to failure modes.
• The final wording avoids causal/statistical overclaiming.