Context
#348 added MediaMessageContent.file_name and populated it end to end, so every
image/document content object now carries the name of the file the user uploaded.
That part is provider-agnostic. The second half — actually rendering the name into
the request payload so the model can read it — was implemented for OpenAI and
Azure OpenAI only, via file_name_text_block() in flo_ai/llm/base_llm.py.
On every other provider the name is carried but never sent, so an agent asked to
report the filename of a document it was given still has no source for it and will
invent a plausible-looking one. That was the original bug #348 set out to fix; it is
only fixed for two of the seven providers.
Current state
| Provider |
Image |
Document |
OpenAI |
done |
done |
AzureOpenAI |
done |
done |
OpenAIVLLM (subclasses OpenAI) |
inherited |
inherited |
Anthropic |
n/a — raises NotImplementedError |
missing |
Gemini |
missing |
missing |
VertexAI (subclasses Gemini) |
missing |
missing |
AWSBedrock |
missing |
missing |
OllamaLLM |
n/a — raises NotImplementedError |
missing |
RootFloLLM |
delegates — inherits whatever the wrapped provider does |
delegates |
Per-provider notes
Anthropic — cleanest of the set, and probably the one to do first. The Messages
API document block has a native title field, so no synthetic text block is needed:
set title in format_document_in_message (anthropic_llm.py:259). Images raise
NotImplementedError, so there's nothing to do there.
Gemini / VertexAI — the real work. format_image_in_message and
format_document_in_message each return a single types.Part, and
gemini_llm.generate appends msg['content'] straight into contents. Emitting a
name Part alongside the media Part means returning multiple Parts, which the content
assembly has to accept. Not a one-liner — worth confirming how the google-genai SDK
handles a nested list in contents before picking an approach.
AWSBedrock — format_image_in_message returns a bare dict rather than a list,
while documents fall through to BaseLLM._rasterize_pdf_to_images, which emits the
OpenAI block shape. Those two shapes are already inconsistent with each other,
independent of file names; worth untangling before adding anything.
Ollama — images raise NotImplementedError; documents use the same inherited
OpenAI-shaped rasterizer as Bedrock, so it has the same question.
Design decision to settle
BaseLLM._rasterize_pdf_to_images deliberately produces the OpenAI Chat Completions
shape and is inherited by Bedrock and Ollama. Adding the name block there would
cover both for free, but it would also mean the base class decides prompt content for
providers that never opted in. #348 avoided that on purpose by overriding
format_document_in_message in OpenAI/AzureOpenAI instead. Whichever way this
goes, it should be a deliberate call rather than a side effect.
Constraint to preserve
The name must ride on the same message as the media, never as a separate message.
An earlier attempt (#282) injected a standalone UserMessage and was removed in #319:
a ForEach over input_filter: [input] counts one item per input message, so one
extra message per file doubles the iterations and runs the per-item pipeline on bare
filename strings.
Acceptance criteria
Out of scope
Sanitising the file name before it enters the prompt (truncation, newline stripping).
It's user-controlled text going into a prompt on every provider, so it deserves its own
issue — but it's a pre-existing property of the approach, not a regression from #348.
Context
#348 added
MediaMessageContent.file_nameand populated it end to end, so everyimage/document content object now carries the name of the file the user uploaded.
That part is provider-agnostic. The second half — actually rendering the name into
the request payload so the model can read it — was implemented for OpenAI and
Azure OpenAI only, via
file_name_text_block()inflo_ai/llm/base_llm.py.On every other provider the name is carried but never sent, so an agent asked to
report the filename of a document it was given still has no source for it and will
invent a plausible-looking one. That was the original bug #348 set out to fix; it is
only fixed for two of the seven providers.
Current state
OpenAIAzureOpenAIOpenAIVLLM(subclassesOpenAI)AnthropicNotImplementedErrorGeminiVertexAI(subclassesGemini)AWSBedrockOllamaLLMNotImplementedErrorRootFloLLMPer-provider notes
Anthropic — cleanest of the set, and probably the one to do first. The Messages
API document block has a native
titlefield, so no synthetic text block is needed:set
titleinformat_document_in_message(anthropic_llm.py:259). Images raiseNotImplementedError, so there's nothing to do there.Gemini / VertexAI — the real work.
format_image_in_messageandformat_document_in_messageeach return a singletypes.Part, andgemini_llm.generateappendsmsg['content']straight intocontents. Emitting aname Part alongside the media Part means returning multiple Parts, which the content
assembly has to accept. Not a one-liner — worth confirming how the google-genai SDK
handles a nested list in
contentsbefore picking an approach.AWSBedrock —
format_image_in_messagereturns a baredictrather than a list,while documents fall through to
BaseLLM._rasterize_pdf_to_images, which emits theOpenAI block shape. Those two shapes are already inconsistent with each other,
independent of file names; worth untangling before adding anything.
Ollama — images raise
NotImplementedError; documents use the same inheritedOpenAI-shaped rasterizer as Bedrock, so it has the same question.
Design decision to settle
BaseLLM._rasterize_pdf_to_imagesdeliberately produces the OpenAI Chat Completionsshape and is inherited by Bedrock and Ollama. Adding the name block there would
cover both for free, but it would also mean the base class decides prompt content for
providers that never opted in. #348 avoided that on purpose by overriding
format_document_in_messageinOpenAI/AzureOpenAIinstead. Whichever way thisgoes, it should be a deliberate call rather than a side effect.
Constraint to preserve
The name must ride on the same message as the media, never as a separate message.
An earlier attempt (#282) injected a standalone
UserMessageand was removed in #319:a
ForEachoverinput_filter: [input]counts one item per input message, so oneextra message per file doubles the iterations and runs the per-item pipeline on bare
filename strings.
Acceptance criteria
on Anthropic, Gemini, VertexAI, Bedrock and Ollama
_formatted_cache) is not mutated, so name blocks can't stackacross nodes and retries — see the regression test in
tests/unit-tests/test_openai_llm.py::test_openai_format_document_includes_file_nameOut of scope
Sanitising the file name before it enters the prompt (truncation, newline stripping).
It's user-controlled text going into a prompt on every provider, so it deserves its own
issue — but it's a pre-existing property of the approach, not a regression from #348.