Skip to content

Images

Add PNG or JPEG images and text to the same Inkling conversation. Common uses include:

  • Image captioning — “Describe this image.”
  • Visual question answering — “How many people are in the image?”
  • Document understanding — “Extract the totals from this receipt.”
  • Image classification — “Which product category best matches this image?”

Use either a native ImagePointer, an OpenAI-compatible image_url part, or a Cookbook ImagePart.

ImagePointer carries the image location, format, width, and height. tml-renderers uses those dimensions to calculate how many image tokens the model input needs.

Native chat messages

Use ImagePointer when you know the local path and dimensions:

from tml_renderers import chat

user = chat.Author(chat.AuthorKind.User)
image = chat.ImagePointer(
    location="photo.png",
    format=chat.ImageFormat.Png,
    width=512,
    height=384,
)
messages = [
    chat.Message(content=chat.Text("Describe this image."), author=user),
    chat.Message(content=image, author=user),
]

The dimensions determine the expected number of image tokens.

OpenAI-compatible Cookbook messages

Use an OpenAI-compatible image_url content part with a base64 data URI:

data_uri = "data:image/png;base64,..."
text_part = {"type": "text", "text": "Describe this image."}
image_part = {"type": "image_url", "image_url": {"url": data_uri}}
messages = [{"role": "user", "content": [text_part, image_part]}]

When given a decodable data URI, tml-renderers derives the image format and dimensions. The Cookbook Inkling renderer accepts these dictionaries directly.

Cookbook-native image parts are supported too:

image_part = {"type": "image", "image": "photo.png"}

The renderer reads the image and constructs the same image input.

Accepted image input

  • Formats: PNG and JPEG.
  • Locations: Local paths, file:// URIs, and base64 data URIs.
  • Native pointers: Provide width and height.
  • OpenAI-compatible data URIs: Dimensions and format are decoded from the image bytes.
  • Cookbook image parts: Accept a local path, data URI, or PIL image.
  • Remote media: HTTP and cloud-storage URLs are not fetched. Download the image in your application first.

Client-side image handling

ImagePointer holds the image location, format, and dimensions. When you build the generation prompt, tml-renderers validates this information and prepares the image input expected by Inkling.

You do not need to generate image patches yourself. Your application supplies valid PNG or JPEG bytes and, for native pointers, the correct width and height.

Run the example

sample_vision.py demonstrates the complete sampling flow:

uv run python -m tinker_cookbook.scripts.inkling.sample_vision
uv run python -m tinker_cookbook.scripts.inkling.sample_vision message_format=chat

Pass image_path=/path/to/image.png to use your own image.

For a complete image-training workflow, including dataset loading, supervised training, and evaluation, use the VLM classifier recipe.

Learn more