docs(weave): add how-to for creating a custom agent signal

docs.json

@@ -568,6 +568,7 @@
                           "weave/guides/tracking/trace-agents-otel",
                           "weave/guides/tracking/view-agent-activity",
                           "weave/guides/tracking/view-agent-signals",
+                          "weave/guides/tracking/create-custom-signal",
                           "weave/guides/tracking/trace-agents-attributes",
                           "weave/guides/tracking/trace-agents-batch",
                           "weave/guides/tracking/trace-sub-agents",

weave/guides/tracking/create-custom-signal.mdx

@@ -0,0 +1,34 @@
+---
+title: "Create a custom signal"
+description: "Create a custom W&B Weave signal to tag agent turns for a behavior you define."
+keywords: ["signals", "custom signal", "agent monitoring", "tags"]
+---
+
+A custom signal scores agent turns against a behavior you define, then shows that result as a tag in the [Agents view](/weave/guides/tracking/view-agent-activity). W&B Weave also provides [built-in signals](/weave/guides/tracking/view-agent-signals) for common cases such as user frustration and jailbreaking. When those built-in signals don't cover what you want to monitor, create a custom signal. This page shows you how to create that custom tag signal in the Weave UI.
+
+## Create a custom tag signal
+
+To create a custom tag signal:
+
+1. Navigate to [wandb.ai](https://wandb.ai/) and select your project.
+1. In the **Weave project sidebar**, click **Agents**.
+1. In the tab bar, click **Signals**.
+1. On the Signals tab, click **+ Create your first signal**.
+1. In the dialog, click **Custom Signal**.
+
+<Frame>
+  ![The Custom tags drawer with Tag name, Definition, Scorer name, and Inference model fields for creating a custom signal.](./imgs/custom-signal.png)
+</Frame>
+
+1. In the **Custom tags** drawer, configure the following fields:
+    - **Tag name**: The label that appears in the Agents view for the behavior you want to monitor.
+    - **Definition**: Describe the behavior the signal should detect.
+    - (Optional) **+ Add tag**: Click to define another tag on the same signal.
+    - **Scorer name**: The display name for this signal.
+    - **Inference model or custom runtime**: The model used to score matching turns. [Serverless Inference](/inference/) is the default. CoreWeave Serverless Inference consumes credits from your W&B account.
+    - (Optional) **Include tool calls**: Expand **Advanced**, then include the turn's tool calls in the context sent to the scoring model.
+    - (Optional) **Only score turns matching**: Expand **Advanced**, then add filters to restrict which turns the signal scores. To score every agent turn, leave this field empty.
+    - (Optional) **Sample rate**: Expand **Advanced**, then set the fraction of matching agent turns the signal scores. To reduce cost on high-traffic agents, we recommend lowering the sample rate.
+1. In the **Custom tags** drawer, click **Create signal**.
+
+After you create the signal, Weave scores matching turns. The tag appears in the Agents view, including on the Signals tab.

weave/guides/tracking/view-agent-signals.mdx

@@ -71,6 +71,8 @@ Each category offers preset templates that you can select directly. Select any c
 
 ### Custom signals
 
+To create a custom tag signal from an empty Signals tab, see [Create a custom signal](/weave/guides/tracking/create-custom-signal).
+
 In the **Create signals** drawer, at the bottom of each of the **Tags** and **Ratings** categories, there is an option to create a custom signal.
 
 To define your own signal, select **Custom Tags** or **Custom Rating**. This opens a configuration screen with the following fields.