Voxtral is Mistral AI’s family of open-weight speech-and-language models, available in two sizes: Voxtral-Mini and Voxtral-Small. The models handle text conversation, audio transcription, combined text-and-audio understanding, and — in the Small variant — function calling driven directly by voice input. This guide walks through self-hosting Voxtral on Amazon SageMaker AI using a bring-your-own-container (BYOC) approach built on the vLLM inference server, following the pattern published on the AWS Machine Learning Blog. In plain terms: instead of relying on a managed model endpoint, the deployment packages the serving stack in a custom Docker container, giving precise control over the vLLM version and the audio libraries Voxtral needs.
Model configuration
The model is configured in code/serving.properties. To deploy Voxtral-Mini, use the following configuration:
option.model_id=mistralai/Voxtral-Mini-3B-2507
option.tensor_parallel_degree=1To deploy Voxtral-Small, use the following configuration:
option.model_id=mistralai/Voxtral-Small-24B-2507
option.tensor_parallel_degree=4Running the accompanying notebook deploys the endpoint and tests it with text, audio, and function-calling capabilities.
Docker container configuration
The full Dockerfile is included in the GitHub repository; the following snippet highlights the key parts:
# Custom vLLM Container for Voxtral Model Deployment on SageMaker
FROM --platform=linux/amd64 vllm/vllm-openai:latest
# Set environment variables for SageMaker
ENV MODEL_CACHE_DIR=/opt/ml/model
ENV TRANSFORMERS_CACHE=/tmp/transformers_cache
ENV HF_HOME=/tmp/hf_home
ENV VLLM_WORKER_MULTIPROC_METHOD=spawn
# Install audio processing dependencies
RUN pip install --no-cache-dir
"mistral_common>=1.8.1"
librosa>=0.10.2
soundfile>=0.12.1
pydub>=0.25.1This Dockerfile extends the official vLLM server image with Voxtral-specific capabilities: mistral_common for tokenization, plus librosa, soundfile, and pydub for audio handling, alongside the SageMaker environment variables needed for model loading and caching. The design separates infrastructure from business logic — the container stays generic while SageMaker dynamically injects the model-specific files (model.py and serving.properties) from Amazon S3 at runtime, so different models can be deployed without rebuilding the container.
Serving properties
The following snippet highlights the key configuration values:
# Model configuration
option.model_id=mistralai/Voxtral-Small-24B-2507
option.tensor_parallel_degree=4
option.dtype=bfloat16
# Voxtral-specific settings (as per official documentation)
option.tokenizer_mode=mistral
option.config_format=mistral
option.load_format=mistral
option.trust_remote_code=true
# Audio processing (Voxtral specifications)
option.limit_mm_per_prompt=audio:8
option.mm_processor_kwargs={"audio_sampling_rate": 16000, "audio_max_length": 1800.0}
# Performance optimizations (vLLM v0.10.0+ features)
option.enable_chunked_prefill=true
option.enable_prefix_caching=true
option.use_v2_block_manager=trueThe configuration follows Mistral’s official recommendations for vLLM deployment: appropriate tokenization modes, audio-processing parameters (up to eight audio files per prompt, with transcription of recordings up to roughly 30 minutes), and vLLM v0.10.0+ performance features such as chunked prefill and prefix caching. Switching between Voxtral-Mini and Voxtral-Small requires only changing the model_id and tensor_parallel_degree parameters.
Custom inference handler
The full inference code lives in model.py in the code folder; the key functions are shown below:
# FastAPI app for SageMaker compatibility
app = FastAPI(title="Voxtral vLLM Inference Server", version="1.1.0")
model_engine = None
# vLLM Server Initialization for Voxtral
def start_vllm_server():
"""Start vLLM server with Voxtral-specific configuration"""
config = load_serving_properties()
cmd = (
"vllm", "serve", config.get("option.model_id"),
"--tokenizer-mode", "mistral",
"--config-format", "mistral",
"--tensor-parallel-size", config.get("option.tensor_parallel_degree"),
"--host", "127.0.0.1",
"--port", "8000"
)
vllm_server_process = subprocess.Popen(cmd, env=vllm_env)
server_ready = wait_for_server()
return server_ready
@app.post("/invocations")
async def invoke_model(request: Request):
"""Handle chat, transcription, and function calling"""
# Transcription requests
if "transcription" in request_data:
audio_source = request_data("transcription")("audio")
return transcribe_audio(audio_source)
# Chat requests with multimodal support
messages = format_messages_for_openai(request_data("messages"))
tools = request_data.get("tools")
# Generate via vLLM OpenAI client
response = openai_client.chat.completions.create(
model=model_config("model_id"),
messages=messages,
tools=tools if supports_function_calling() else None
)
return responseThe handler creates a FastAPI-based server that integrates directly with vLLM. It processes multimodal content — both Base64-encoded audio and audio URLs — loads model configuration dynamically from serving.properties, and supports function calling for Voxtral-Small deployments.
Deployment notebook
The included notebook in the voxtral-vllm-byoc folder organizes the entire deployment process for both model variants:
import boto3
import sagemaker
from sagemaker.model import Model
# Initialize SageMaker session
sagemaker_session = sagemaker.Session()
role = sagemaker.get_execution_role()
bucket = "your-s3-bucket"
# Upload model artifacts to S3
byoc_config_uri = sagemaker_session.upload_data(
path="./code",
bucket=bucket,
key_prefix="voxtral-vllm-byoc/code"
)
# Configure custom container image
account_id = boto3.client('sts').get_caller_identity()('Account')
region = boto3.Session().region_name
image_uri = f"{account_id}.dkr.ecr.{region}.amazonaws.com/voxtral-vllm-byoc:latest"
# Create SageMaker model
voxtral_model = Model(
image_uri=image_uri,
model_data={
"S3DataSource": {
"S3Uri": f"{byoc_config_uri}/",
"S3DataType": "S3Prefix",
"CompressionType": "None"
}
},
role=role,
env={
'MODEL_CACHE_DIR': '/opt/ml/model',
'TRANSFORMERS_CACHE': '/tmp/transformers_cache',
'SAGEMAKER_BIND_TO_PORT': '8080'
}
)
# Deploy to endpoint
predictor = voxtral_model.deploy(
initial_instance_count=1,
instance_type="ml.g6.12xlarge", # For Voxtral-Small
container_startup_health_check_timeout=1200,
wait=True
)Model use cases
Voxtral supports a range of text and speech-to-text use cases, and Voxtral-Small adds tool use with voice input. The GitHub repository contains complete code; the snippets below cover each supported scenario.
Text only
A basic text interaction — the user sends a text query and receives a structured response:
payload = {
"messages": (
{
"role": "user",
"content": "Hello! Can you tell me about the advantages of using vLLM for model inference?"
}
),
"max_tokens": 200,
"temperature": 0.2,
"top_p": 0.95
}
response = predictor.predict(payload)Transcription only
Speech-to-text transcription with temperature set to 0 for deterministic output. The model accepts an audio file URL or Base64-encoded audio and returns the transcript without additional interpretation:
payload = {
"transcription": {
"audio": "https://audiocdn.frenchtoday.com/file/ft-public-files/audiobook-samples/AMPFE/AMP%20FE%20Ch%2002%20Story%20Slower.mp3",
"language": "fr",
"temperature": 0.0
}
}
response = predictor.predict(payload)Text and audio understanding
Combining text instructions with audio input enables guided transcription and audio-analysis tasks — the model follows text commands while analyzing the supplied audio:
payload = {
"messages": (
{
"role": "user",
"content": (
{
"type": "text",
"text": "Can you summarise this audio file"
},
{
"type": "audio",
"path": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama.mp3"
}
)
}
),
"max_tokens": 300,
"temperature": 0.2,
"top_p": 0.95
}
response = predictor.predict(payload)Tool use
Function calling lets the model interpret voice commands and execute predefined tools. This example handles weather queries from voice input, with the model selecting the appropriate function and returning structured results:
# Define weather tool configuration
WEATHER_TOOL = {
"type": "function",
"function": {
"name": "get_current_weather",
"description": "Get the current weather for a specific location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA"
},
"format": {
"type": "string",
"enum": ("celsius", "fahrenheit"),
"description": "The temperature unit to use."
}
},
"required": ("location", "format")
}
}
}
# Mock weather function
def mock_weather(location, format="celsius"):
"""Always returns sunny weather at 25°C/77°F"""
temp = 77 if format.lower() == "fahrenheit" else 25
unit = "°F" if format.lower() == "fahrenheit" else "°C"
return f"It's sunny in {location} with {temp}{unit}"
# Test payload with audio
payload = {
"messages": (
{
"role": "user",
"content": (
{
"type": "audio",
"path": "https://huggingface.co/datasets/patrickvonplaten/audio_samples/resolve/main/fn_calling.wav"
}
)
}
),
"temperature": 0.2,
"top_p": 0.95,
"tools": (WEATHER_TOOL)
}
response = predictor.predict(payload)Strands Agents integration
Voxtral can also be integrated with the open-source Strands Agents SDK to build agents that select and execute tools — calculators, file operations, or shell commands from the prebuilt toolset — based on user queries, enabling multi-step workflows through natural language:
# SageMaker integration with Strands agents
# from strands import Agent
from strands import Agent
from strands.models.sagemaker import SageMakerAIModel
from strands_tools import calculator, current_time, file_read, shell
model = SageMakerAIModel(
endpoint_config={
"endpoint_name": endpoint_name,
"region_name": "us-west-2",
},
payload_config={
"max_tokens": 1000,
"temperature": 0.7,
"stream": False,
}
)
agent = Agent(model=model, tools=(calculator, current_time, file_read, shell))
response = agent("What is the square root of 12?")Clean up
After experimenting, delete the SageMaker endpoints created in the notebook to avoid unnecessary costs:
# Delete SageMaker endpoint
print(f" Deleting endpoint: {endpoint_name}")
predictor.delete_endpoint(delete_endpoint_config=True)
print(" Endpoint deleted successfully")Conclusion, limitations, and what to watch
This walkthrough shows how to self-host Voxtral on SageMaker with a BYOC approach: a production-oriented setup using the vLLM framework and official Voxtral optimizations for both the Mini and Small variants, covering text conversation, audio transcription, multimodal understanding, and voice-driven function calling.
A few practical caveats apply. GPU-backed SageMaker instances are billed while the endpoint is running, so idle endpoints are a common source of unexpected cost — AWS’s guidance recommends instance families such as ml.g6 for these models, and right-sizing matters. Version pinning is important: the configuration targets vLLM 0.10.0+, and both vLLM and mistral_common evolve quickly, so future library versions may require configuration changes. Self-hosting also shifts responsibility for scaling, monitoring, and security patching onto the deploying team compared with a fully managed API. For adjacent reading on this site, see the Mistral OCR 3 technical review and structured output on Amazon Bedrock.
