Monitor your AI application for issues

This guide shows you how to monitor your AI application's issue behavior using the Live Dashboard and Conversations view.

Use this feature to monitor issues in your RAG application or agent as they happen. You can track metrics over time to catch regressions or verify improvements. When issues arise, drill down into specific conversations to perform error analysis.

Prerequisites

• A Blue Guardrails account with a workspace
• A workspace role of Viewer or higher (see Roles and permissions)
• An AI application sending traces to Blue Guardrails (see Getting started)

Check your issue rate

Open your workspace and click Dashboard in the sidebar.

Blue Guardrails evaluates assistant messages for issues. Each message can contain multiple issues. The issue rate is the percentage of evaluated messages that have at least one issue.

The stats band at the top shows key metrics:

• Issue rate - percentage of evaluated messages with at least one issue
• Total issues - total count of detected issues across all messages
• Evaluated messages - number of assistant messages analyzed

If you need a different time range, use the time selector in the top right. You can choose from 15 minutes to 14 days.

Identify problematic messages

Look at the waffle chart below the stats band. Each cell represents a message or group of messages.

• Red cells indicate messages with issues.
• Click a cell to see the messages it represents.

Check the Issue types chart on the right to see which categories appear most often.

Scroll to the bottom of the page to see a list of messages as they come in. Click any message to open it in the Conversations view.

Review specific conversations

1. Click Conversations in the sidebar.
2. If you want to narrow down results, filter by date range using the date picker.
3. Browse the conversation list in the left panel.
4. Select a conversation to view its messages in the right panel.
5. Look for pink underlined text in assistant messages. These are detected issues.
6. Hover over an annotation to see:
   • The issue type (e.g., "Fabrication")
   • An explanation of why it was flagged

Verify an issue

If you want to check whether flagged text is actually an issue, use the search box to find that text in earlier messages. If the text doesn't appear in user messages or tool results, the assistant fabricated it.

1. Select a conversation with a flagged issue.
2. Note the text marked as an issue.
3. Use the search box above the messages to search for that text.
4. Check if it appears in any source messages (user input, tool results, or system prompts).