Skip to content

Original file name only reaches the model on OpenAI/Azure — missing for Anthropic, Gemini, Bedrock, Ollama #349

Description

@rootflo-hardik

Context

#348 added MediaMessageContent.file_name and populated it end to end, so every
image/document content object now carries the name of the file the user uploaded.

That part is provider-agnostic. The second half — actually rendering the name into
the request payload so the model can read it — was implemented for OpenAI and
Azure OpenAI only
, via file_name_text_block() in flo_ai/llm/base_llm.py.

On every other provider the name is carried but never sent, so an agent asked to
report the filename of a document it was given still has no source for it and will
invent a plausible-looking one. That was the original bug #348 set out to fix; it is
only fixed for two of the seven providers.

Current state

Provider Image Document
OpenAI done done
AzureOpenAI done done
OpenAIVLLM (subclasses OpenAI) inherited inherited
Anthropic n/a — raises NotImplementedError missing
Gemini missing missing
VertexAI (subclasses Gemini) missing missing
AWSBedrock missing missing
OllamaLLM n/a — raises NotImplementedError missing
RootFloLLM delegates — inherits whatever the wrapped provider does delegates

Per-provider notes

Anthropic — cleanest of the set, and probably the one to do first. The Messages
API document block has a native title field, so no synthetic text block is needed:
set title in format_document_in_message (anthropic_llm.py:259). Images raise
NotImplementedError, so there's nothing to do there.

Gemini / VertexAI — the real work. format_image_in_message and
format_document_in_message each return a single types.Part, and
gemini_llm.generate appends msg['content'] straight into contents. Emitting a
name Part alongside the media Part means returning multiple Parts, which the content
assembly has to accept. Not a one-liner — worth confirming how the google-genai SDK
handles a nested list in contents before picking an approach.

AWSBedrockformat_image_in_message returns a bare dict rather than a list,
while documents fall through to BaseLLM._rasterize_pdf_to_images, which emits the
OpenAI block shape. Those two shapes are already inconsistent with each other,
independent of file names; worth untangling before adding anything.

Ollama — images raise NotImplementedError; documents use the same inherited
OpenAI-shaped rasterizer as Bedrock, so it has the same question.

Design decision to settle

BaseLLM._rasterize_pdf_to_images deliberately produces the OpenAI Chat Completions
shape and is inherited by Bedrock and Ollama. Adding the name block there would
cover both for free, but it would also mean the base class decides prompt content for
providers that never opted in. #348 avoided that on purpose by overriding
format_document_in_message in OpenAI/AzureOpenAI instead. Whichever way this
goes, it should be a deliberate call rather than a side effect.

Constraint to preserve

The name must ride on the same message as the media, never as a separate message.
An earlier attempt (#282) injected a standalone UserMessage and was removed in #319:
a ForEach over input_filter: [input] counts one item per input message, so one
extra message per file doubles the iterations and runs the per-item pipeline on bare
filename strings.

Acceptance criteria

  • A named document/image produces a payload the model can read the file name from,
    on Anthropic, Gemini, VertexAI, Bedrock and Ollama
  • Message count is unchanged — the name never becomes its own message
  • Unnamed media produces byte-identical payloads to today
  • Cached formatting (_formatted_cache) is not mutated, so name blocks can't stack
    across nodes and retries — see the regression test in
    tests/unit-tests/test_openai_llm.py::test_openai_format_document_includes_file_name
  • Per-provider unit tests mirroring the three added in fix: carry the original file name through to media inputs #348

Out of scope

Sanitising the file name before it enters the prompt (truncation, newline stripping).
It's user-controlled text going into a prompt on every provider, so it deserves its own
issue — but it's a pre-existing property of the approach, not a regression from #348.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions