Detailed description of the requested feature
The experimental OpenAI-compatible server currently supports VLM image inputs only when the image is provided via a local file path. This prevents clients that use the standard OpenAI vision API format from sending images as base64-encoded data URLs.
I would like to add support for base64-encoded images, following the OpenAI-compatible image_url format, e.g. data:image/jpeg;base64,... or data:image/png;base64,....
The server should decode the base64 image data and pass it through the existing VLM image preprocessing/inference pipeline, while retaining the current file-path support for backwards compatibility.
This would make the experimental server more compatible with existing OpenAI-compatible clients and applications, which commonly transmit images as base64 data URLs rather than exposing local filesystem paths. It would also be useful for deployments where the client and inference server do not share the same filesystem.
My implementation is available at: https://github.com/FABallemand/TensorRT-Edge-LLM/tree/feat/server_base64_support
Per CONTRIBUTING.md, I am opening this issue for approval before requesting code review.
Timeline
No hard deadline.
Describe alternatives you've considered
The current workaround is to save the image to a file accessible by the server and provide its path in the request. This adds unnecessary filesystem I/O and requires the client and server to have access to the same filesystem.
Adding native base64 image support to the server would provide a more interoperable solution while preserving the existing file-path behavior.
Target hardware/use case
Jetson Thor (SM_110, JetPack 7.1, TensorRT 10.13.3.9), using the experimental OpenAI-compatible server to host VLMs.
Detailed description of the requested feature
The experimental OpenAI-compatible server currently supports VLM image inputs only when the image is provided via a local file path. This prevents clients that use the standard OpenAI vision API format from sending images as base64-encoded data URLs.
I would like to add support for base64-encoded images, following the OpenAI-compatible
image_urlformat, e.g.data:image/jpeg;base64,...ordata:image/png;base64,....The server should decode the base64 image data and pass it through the existing VLM image preprocessing/inference pipeline, while retaining the current file-path support for backwards compatibility.
This would make the experimental server more compatible with existing OpenAI-compatible clients and applications, which commonly transmit images as base64 data URLs rather than exposing local filesystem paths. It would also be useful for deployments where the client and inference server do not share the same filesystem.
My implementation is available at: https://github.com/FABallemand/TensorRT-Edge-LLM/tree/feat/server_base64_support
Per
CONTRIBUTING.md, I am opening this issue for approval before requesting code review.Timeline
No hard deadline.
Describe alternatives you've considered
The current workaround is to save the image to a file accessible by the server and provide its path in the request. This adds unnecessary filesystem I/O and requires the client and server to have access to the same filesystem.
Adding native base64 image support to the server would provide a more interoperable solution while preserving the existing file-path behavior.
Target hardware/use case
Jetson Thor (SM_110, JetPack 7.1, TensorRT 10.13.3.9), using the experimental OpenAI-compatible server to host VLMs.