diff --git a/docs.json b/docs.json index 56f929df18..468b10cf58 100644 --- a/docs.json +++ b/docs.json @@ -568,6 +568,7 @@ "weave/guides/tracking/trace-agents-otel", "weave/guides/tracking/view-agent-activity", "weave/guides/tracking/view-agent-signals", + "weave/guides/tracking/create-custom-signal", "weave/guides/tracking/trace-agents-attributes", "weave/guides/tracking/trace-agents-batch", "weave/guides/tracking/trace-sub-agents", diff --git a/weave/guides/tracking/create-custom-signal.mdx b/weave/guides/tracking/create-custom-signal.mdx new file mode 100644 index 0000000000..9af768f1b0 --- /dev/null +++ b/weave/guides/tracking/create-custom-signal.mdx @@ -0,0 +1,34 @@ +--- +title: "Create a custom signal" +description: "Create a custom W&B Weave signal to tag agent turns for a behavior you define." +keywords: ["signals", "custom signal", "agent monitoring", "tags"] +--- + +A custom signal scores agent turns against a behavior you define, then shows that result as a tag in the [Agents view](/weave/guides/tracking/view-agent-activity). W&B Weave also provides [built-in signals](/weave/guides/tracking/view-agent-signals) for common cases such as user frustration and jailbreaking. When those built-in signals don't cover what you want to monitor, create a custom signal. This page shows you how to create that custom tag signal in the Weave UI. + +## Create a custom tag signal + +To create a custom tag signal: + +1. Navigate to [wandb.ai](https://wandb.ai/) and select your project. +1. In the **Weave project sidebar**, click **Agents**. +1. In the tab bar, click **Signals**. +1. On the Signals tab, click **+ Create your first signal**. +1. In the dialog, click **Custom Signal**. + + + ![The Custom tags drawer with Tag name, Definition, Scorer name, and Inference model fields for creating a custom signal.](./imgs/custom-signal.png) + + +1. In the **Custom tags** drawer, configure the following fields: + - **Tag name**: The label that appears in the Agents view for the behavior you want to monitor. + - **Definition**: Describe the behavior the signal should detect. + - (Optional) **+ Add tag**: Click to define another tag on the same signal. + - **Scorer name**: The display name for this signal. + - **Inference model or custom runtime**: The model used to score matching turns. [Serverless Inference](/inference/) is the default. CoreWeave Serverless Inference consumes credits from your W&B account. + - (Optional) **Include tool calls**: Expand **Advanced**, then include the turn's tool calls in the context sent to the scoring model. + - (Optional) **Only score turns matching**: Expand **Advanced**, then add filters to restrict which turns the signal scores. To score every agent turn, leave this field empty. + - (Optional) **Sample rate**: Expand **Advanced**, then set the fraction of matching agent turns the signal scores. To reduce cost on high-traffic agents, we recommend lowering the sample rate. +1. In the **Custom tags** drawer, click **Create signal**. + +After you create the signal, Weave scores matching turns. The tag appears in the Agents view, including on the Signals tab. diff --git a/weave/guides/tracking/imgs/custom-signal.png b/weave/guides/tracking/imgs/custom-signal.png new file mode 100644 index 0000000000..517eb58c47 Binary files /dev/null and b/weave/guides/tracking/imgs/custom-signal.png differ diff --git a/weave/guides/tracking/view-agent-signals.mdx b/weave/guides/tracking/view-agent-signals.mdx index e8d6301d1e..984c3fcd35 100644 --- a/weave/guides/tracking/view-agent-signals.mdx +++ b/weave/guides/tracking/view-agent-signals.mdx @@ -71,6 +71,8 @@ Each category offers preset templates that you can select directly. Select any c ### Custom signals +To create a custom tag signal from an empty Signals tab, see [Create a custom signal](/weave/guides/tracking/create-custom-signal). + In the **Create signals** drawer, at the bottom of each of the **Tags** and **Ratings** categories, there is an option to create a custom signal. To define your own signal, select **Custom Tags** or **Custom Rating**. This opens a configuration screen with the following fields.