Skip to content

Audio

Inkling natively supports audio input, which you can combine with other modalities such as text and tool calls in the same conversation. Common uses include:

  • Speech transcription (ASR) — “Transcribe this speech exactly.”
  • Audio question answering — “What topic are the speakers discussing?”
  • Speaking-style classification — “Is the speaker whispering, laughing, or speaking normally?”
  • Information extraction — “List the names and dates mentioned in this recording.”

The Cookbook audio recipes demonstrate complete ASR and speaking-style workflows, including sampling, supervised fine-tuning, reinforcement learning, and evaluation.

Use either a native AudioPointer or an OpenAI-compatible input_audio part. tml-renderers decodes and DMel-encodes the audio on your client before the model request.

Native chat messages

Use AudioPointer to refer to a local file:

from tml_renderers import chat

user = chat.Author(chat.AuthorKind.User)
audio = chat.AudioPointer(
    location="speech.wav",
    format=chat.AudioFormat.Wav,
    num_frames=48_000,
    sample_rate=16_000,
)
messages = [
    chat.Message(content=chat.Text("Transcribe this."), author=user),
    chat.Message(content=audio, author=user),
]

num_frames is the number of source audio frames—not the number of DMel tokens. sample_rate describes the source file.

OpenAI-compatible Cookbook messages

For OpenAI-compatible messages, put base64-encoded audio in an input_audio content part:

audio_base64 = "..."  # Base64-encoded WAV bytes
audio_part = {
    "type": "input_audio",
    "input_audio": {"data": audio_base64, "format": "wav"},
}
text_part = {"type": "text", "text": "Transcribe this."}
messages = [{"role": "user", "content": [text_part, audio_part]}]

For WAV input, tml-renderers reads the frame count and sample rate from the WAV header. MP3 and FLAC inputs must include num_frames and sample_rate:

{
    "type": "input_audio",
    "input_audio": {
        "data": audio_base64,
        "format": "mp3",
        "num_frames": 48_000,
        "sample_rate": 24_000,
    },
}

The Cookbook Inkling renderer accepts these dictionaries directly.

Accepted audio input

  • Formats: WAV, MP3, and FLAC.
  • Locations: Local file paths for native AudioPointer; base64 audio for OpenAI-compatible input_audio.
  • WAV metadata: Frame count and sample rate can be read from a valid WAV header.
  • MP3 and FLAC metadata: Provide num_frames and sample_rate.
  • Remote media: HTTP and cloud-storage URLs are not fetched. Download the file in your application first.

The encoding is eager and client-side. By the time build_generation_prompt(...) returns, the audio is ready to send to Inkling.

Client-side audio encoding

AudioPointer holds the source location and metadata. When you build the generation prompt, tml-renderers:

  1. Reads the local or inline audio.
  2. Decodes and resamples it.
  3. Encodes it as quantized mel features (DMels).
  4. Places those features into the model-ready prompt.

You do not need to compute DMels yourself. Your application supplies audio bytes plus the correct format and metadata.

Preparing speech input

Mono 16 kHz WAV is a simple interchange format for speech datasets. Other accepted source rates are decoded and resampled by the client encoder.

Run the example

sample_audio.py demonstrates the complete sampling flow:

uv run python -m tinker_cookbook.scripts.inkling.sample_audio
uv run python -m tinker_cookbook.scripts.inkling.sample_audio message_format=chat

Pass audio_path=/path/to/audio.wav to use your own file.

Learn more