Adapt issue detection to your use case

Create an evaluator to define what Blue Guardrails checks and when it runs. Evaluators contain issue types, domain context, and run settings.

A pharmaceutical application might check for dosage_error and contraindication_omission. A legal application might use citation_missing and precedent_misattribution.

Prerequisites

• A Blue Guardrails account with a workspace
• A workspace role of Contributor or higher (see Roles and permissions)
• Traced conversations in your workspace to test the evaluator

Open the evaluator list

Click Evaluators in the sidebar.

Each workspace has a built-in Issue Evaluator, marked Default. The active evaluator runs automatically on future messages that match its run condition.

You can edit the built-in evaluator or create another one. The actions menu also lets you duplicate an evaluator.

Only one evaluator can be active in a workspace. You can still select inactive evaluators in experiments and live detection requests.

Create an evaluator

1. Click New Evaluator.
2. On the Info tab, enter a name and an optional description.
3. Configure the remaining tabs.
4. Click Save.

To start with the built-in settings, open the actions menu for Issue Evaluator and click Duplicate. Then open the copy to rename and edit it.

Set when the evaluator runs

Open the Run Configuration tab.

Status

Turn on Status if this evaluator should run automatically on future matching messages.

If another evaluator is active, save the new evaluator as inactive. Deactivate the current evaluator before activating the new one.

Scope

Choose which messages the evaluator may annotate during each run:

• Only the triggering message annotates the message that matched the run condition. Earlier messages remain available as context.
• All messages since the last run annotates new messages since this evaluator last ran. The first run can annotate the conversation up to the triggering message.

Run condition

Use the filter controls to select which incoming messages can trigger the evaluator. You must select at least one message role.

The default condition runs on assistant messages and excludes Blue Guardrails system conversations. Other filters include generation model, tool name, origin run tags, and conversation tags.

Sample percentage

Set the percentage of matching conversations to evaluate during live trace processing. Use 100 to evaluate every matching conversation.

Add domain context

Open the Context tab and enter any background information the evaluator needs. Include rules or terminology that affects whether content is an issue.

Example for a pharmaceutical assistant:

This is a medical Q&A assistant that answers from pharmaceutical SmPC documents. Flag incorrect drug dosages and invented medical conditions. Flag studies that aren't in the provided context and side effects attributed to the wrong medication. Check dosage units, administration frequency, and contraindications against the source.

Configure issue types

Open the Issue Types tab. Each issue type needs a name and description.

1. Click Add Issue Type.
2. Enter the issue type name.
3. Enter a description that states when the issue type applies.
4. Repeat for each issue type you need.
5. Remove issue types with the trash icon.

Example:

Issue type: dosage_error

Description: The response gives a medication dosage, unit, administration frequency, or maximum daily limit that conflicts with the provided source.

Use the same name for the same issue type. Describe what evidence the evaluator should compare.

Activate the evaluator

If you saved the evaluator as inactive:

1. Return to the evaluator list.
2. Deactivate the current active evaluator.
3. Activate the evaluator you want to use.

Activation affects future automatic evaluations. Existing evaluations and annotations don't change.

Test the evaluator

Test the evaluator before relying on it for monitoring:

1. Send a new trace that matches the run condition. You can instead run an experiment or select the evaluator in a live detection request.
2. Open the result in Live, Conversations, or the experiment. For live detection, inspect the response body.
3. Check the issue types and annotated text spans.

If the results are too broad or narrow, revise the domain context and issue type descriptions. If no evaluation appears, check the run condition and sample percentage.