Detect issues from multimodal input

Send traces that include images or PDF files in user messages and get issue detection on the model's responses.

This guide shows you how to send traces that contain images or PDF files to Blue Guardrails. Issue detection works on conversations with multimodal input the same way it works on text-only conversations. You instrument your code, send traces, and detection runs automatically.

Prerequisites

A Blue Guardrails account with a workspace and credits
A workspace role of Contributor or higher (see Roles and permissions)
A workspace API key with the Write traces permission (see Create an API key for your workspace)
An application that sends images or PDF files to an LLM (see Getting started for basic setup)

Send traces with image input

Your instrumentation setup stays the same as described in Getting started. The only change is in the messages you send to the model: include one or more images alongside text.

Each framework has its own format for image content. The examples below load an image from disk and send it as part of a user message.

import os
import base64
from pathlib import Path

os.environ['OTEL_EXPORTER_OTLP_TRACES_ENDPOINT'] = 'https://api.blueguardrails.com/v1/traces'
os.environ['OTEL_EXPORTER_OTLP_HEADERS'] = (
    f'Authorization=Bearer {YOUR_API_KEY}'
)

import openai
import logfire

client = openai.Client()
logfire.configure(send_to_logfire=False)
logfire.instrument_openai()

image_b64 = base64.b64encode(Path('chart.png').read_bytes()).decode()

response = client.chat.completions.create(
    model='gpt-5.4-mini',
    messages=[
        {'role': 'system', 'content': 'Answer the question based on the provided chart.'},
        {
            'role': 'user',
            'content': [
                {'type': 'text', 'text': 'What does this chart show?'},
                {
                    'type': 'image_url',
                    'image_url': {'url': f'data:image/png;base64,{image_b64}'},
                },
            ],
        },
    ],
)
print(response.choices[0].message.content)

You can send multiple images in a single message. Add more image content blocks to the same user message.

Send traces with PDF file input

PDF file tracing is available for OpenAI Responses API input_file blocks. Use inline base64 file_data when you want the PDF to appear in Blue Guardrails.

Set OPENAI_API_KEY and BLUE_GUARDRAILS_API_KEY in your environment.

Put sample.pdf in the same directory as the example file.

Create a file called responses_file.py and paste the following:

import base64
import os
from pathlib import Path

api_key = os.environ["BLUE_GUARDRAILS_API_KEY"]
headers = f"Authorization=Bearer {api_key}"

os.environ["OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"] = "https://api.blueguardrails.com/v1/traces"
os.environ["OTEL_EXPORTER_OTLP_HEADERS"] = headers

import logfire
from openai import OpenAI

logfire.configure(send_to_logfire=False)
logfire.instrument_openai()

pdf_base64 = base64.b64encode(Path("sample.pdf").read_bytes()).decode("ascii")

client = OpenAI()
response = client.responses.create(
    model="gpt-5.4-mini",
    instructions="Answer the user's question using the attached PDF.",
    input=[
        {
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "What revenue number is mentioned in the attached PDF?",
                },
                {
                    "type": "input_file",
                    "filename": "sample.pdf",
                    "file_data": f"data:application/pdf;base64,{pdf_base64}",
                },
            ],
        }
    ],
)

print(response.output_text)

The file block uses three fields for tracing:

Set type to input_file.
Use a .pdf extension in filename.
Set file_data to a data:application/pdf;base64,... URI.

A request that only contains an OpenAI file_id doesn't include the file bytes in the trace. Use file_data if you want Blue Guardrails to receive the PDF. Issue detection can only run, if Blue Guardrails receives the PDF.

Review detected issues

After running your code, traces appear in your workspace within a few seconds. Blue Guardrails extracts images and PDF files from the trace and evaluates the assistant's response against them for issues.

Open Conversations in the sidebar and select a conversation. Images and PDF files appear inline in user messages. Click a PDF or image preview to open a larger viewer. Issues in the assistant's response are underlined in pink, just like in text-only conversations.

Multimodal issue detection in the conversations view

For more on working with the conversations view, see Monitor your AI application for issues.

Detect issues from multimodal input

Prerequisites

Send traces with image input

Send traces with PDF file input

Review detected issues

On this page