docs(weave): add how-to for creating a custom agent signal

[anastasiaguspan]

Summary

• Adds a short how-to for creating a custom tag signal from an empty Agents Signals tab, including the Custom tags drawer fields from the product screenshot.
• Places the page immediately after Monitor your agents with signals in English navigation.
• Cross-links the existing Custom signals subsection to the new page.

Test plan

• Confirm the + Create your first signal → Custom Signal dialog labels against the live UI (the screenshot shows the Custom tags drawer after that step).
• Confirm field captions in the Custom tags drawer: Tag name, Definition, Scorer name, Inference model or custom runtime, and the Advanced options.
• Confirm the new tag appears in the Agents view / Signals tab after create.
• Confirm English nav order: view-agent-signals, then create-custom-signal.
• Spot-check the screenshot renders on the local Mintlify preview (mint dev).

Made with Cursor

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
wandb	🟢 Ready	View Preview	Aug 20, 2026, 5:47 PM

---

📄 Preview this PR's docs · a79f1a2

Page	Change
Create Custom Signal	Added · +34 −0
View Agent Signals	Changed · +2 −0

No preview available (1)

File	Change
docs.json	Changed · +1 −0

_{Updated 2026-08-20 17:48:29 UTC for deployment.}

[anastasiaguspan]

Sources and decision log

Sources

• SME draft + screenshot (/Users/aguspan/Desktop/custom-signal.png, copied to weave/guides/tracking/imgs/custom-signal.png) — authoritative for the empty-state Custom tags drawer and field captions. Confidence: high for visible UI; the Custom Signal dialog is in the SME draft but not in the screenshot.
• Existing docs weave/guides/tracking/view-agent-signals.mdx — authoritative for presets, ratings, and the already-have-signals Create signals drawer. Field names there (Prompt template, Scorer prompt) do not match this screenshot (Tag name, Definition). This page trusts the screenshot for the empty-state custom-tag flow.
• Learn Weave (Confluence, updated 2026-08-18): custom signals use your own prompt, filters, and sample rate; scoring uses Serverless Inference. https://coreweave.atlassian.net/wiki/spaces/~7120207e2a07951dcd4684b4fa108b9a0f4262/pages/2107573399
• Notion Weave for Agents Product and Signals Private Preview — Customer Guide — useful for product intent; the customer guide is monitors-era UI and was treated as stale for this drawer.
• No WBDOCS ticket found. Sourcegraph was unavailable; Glean GitHub code search returned no indexed hits for this UI.

Decisions

• Doc type: How-to, not a concept explainer. The Signals overview already explains what signals are.
• Scope: Empty-state custom tag flow only. Custom ratings and preset templates stay on the overview page.
• Title: “Create a custom signal” (task-oriented, matches the SME draft). Procedure H2 is “Create a custom tag signal” because the screenshot is the Custom tags drawer.
• Placement: English nav immediately after view-agent-signals. Localized fr/ ja/ ko/ nav was not updated (translation is a separate process).
• UI labels: Screenshot captions win over the overview page where they disagree. Submit action is Create signal (screenshot), not “hit enter” (draft).
• Cross-link: Added a pointer from the overview Custom signals subsection; did not delete or rewrite that subsection’s Create-signals-drawer fields.

Intentionally omitted

• Custom rating creation (not in the screenshot).
• The already-have-signals + New signal path (documented on the overview page).
• Prerequisites such as inference credits or required agent traces (not in the SME draft; flagged below).
• Localized pages.

Needs SME verification

• Dialog after + Create your first signal: draft says Custom Signal; screenshot only shows the Custom tags drawer already open. Confirm the exact dialog label and whether this step exists.
• Confirm + Create your first signal is empty-state-only, versus + New signal when signals already exist.
• Confirm field captions: Tag name, Definition, Scorer name, Inference model or custom runtime, Include tool calls, Only score turns matching, Sample rate.
• Confirm Tag name is what appears in the Agents view / Signals tab.
• Confirm custom ratings should stay off this page.
• Overview page Custom signals fields (Prompt template, Scorer prompt) appear to describe a different UI than this drawer — confirm whether both UIs exist or the overview is stale.
• Official product name: Serverless Inference vs CoreWeave Serverless Inference. This page matches the sibling Signals page and the screenshot credit caption.
• After Create signal, what should the reader see (drawer closes, signal listed, delay before tags appear)?
• Missing prerequisites: W&B account, project with agent turns, Serverless Inference credits.

[anastasiaguspan]

Resume prompt

Continue work on #3089 (branch aguspan/docs-create-custom-signal). No JIRA ticket.

This PR adds weave/guides/tracking/create-custom-signal.mdx, a screenshot at weave/guides/tracking/imgs/custom-signal.png, English nav placement immediately after view-agent-signals, and a cross-link from that page’s Custom signals subsection.

Key decisions already made:

• How-to for the empty-state custom tag flow only.
• Trust the screenshot for Custom tags drawer fields; the SME draft’s Custom Signal dialog step is included but unverified in the screenshot.
• Do not rewrite the overview page’s Create-signals-drawer custom-signal fields; only add a pointer.

If you pick this up: resolve the SME checklist on the Sources and decision log comment, especially dialog label, empty-state vs + New signal, and whether the overview’s Prompt template / Scorer prompt UI is a second flow or stale docs.

[anastasiaguspan]

{
  "skill": "author-docs",
  "version": "1.2.0",
  "depends_on": {
    "style-guide": "1.0.0",
    "ui-procedure": "1.0.0"
  },
  "skills_invoked": ["style-guide", "ui-procedure"],
  "model": "Cursor Grok 4.6",
  "mode": "autonomous"
}

[wandb-docs-pr-writer]

Images automagically compressed by Calibre's image-actions ✨

Compression reduced images by 78.4%, saving 140.0 KB.

Filename	Before	After	Improvement	Visual comparison
weave/guides/tracking/imgs/custom-signal.png	178.6 KB	38.6 KB	78.4%	View diff

[github-actions]

Readability impact

Word-weighted Flesch-Kincaid grade change across 1 changed page: +0.0 (unchanged).

Lower Flesch-Kincaid grade and higher reading ease both mean easier to read. This check is informational and never blocks a PR.

Human readability

Page	FK grade before	FK grade after	FK Δ	Reading ease Δ	Direction
weave/guides/tracking/create-custom-signal.mdx	—	6.7	—	—	n/a
weave/guides/tracking/view-agent-signals.mdx	8.2	8.2	+0.0	+0.1	unchanged

AI agent comprehension

Rated 0-3 (higher is easier for an agent to parse and act on).

Page	Before	After	Δ
weave/guides/tracking/view-agent-signals.mdx	3	3	+0
weave/guides/tracking/create-custom-signal.mdx	—	3	—

_{Curated-docs baseline median FK grade by type: conceptual 10.5, procedural 8.8, reference 9.4.}

From workflow run 32399309787

[github-actions]

🔗 Link Checker Results

✅ All links are valid!

No broken links were detected.

Preview: https://wb-21fd5541-aguspan-docs-create-custom-signal.mintlify.site

[anastasiaguspan]

this was my prompt (in cursor) (explicit about skill because of local testing):
use author-docs to create a new short page for the following. in my Desktop folder use screenshot /Users/aguspan/Desktop/custom-signal.png. put the page after the view-agent-signals page. for the following content draft:
Create a custom signal

Weave provides many built-in signals that monitor for common types of agentic application interests, such a frustrated users or jailbreaking. But you can also easily create a signal for any custom need for your application.

Under Agents, pick the Signals tab
Click Create your first signal and pick “custom Signal” in the dialog box.

This is where you can create a custom signal,
Enter the fields shown and hit enter. Create the tag to be what you want shown in the agents dashboard (what it is that you want to monitor).

[w-b-hivemind]

HiveMind Sessions

1 session · 19m · $0.00

Session	Agent	Duration	Tokens	Cost	Lines
Create Custom Signal Documentation Page	cursor	19m	4.2M	$0.00	+64 -27
Total		19m	4.2M	$0.00	+64 -27

View all sessions in HiveMind →

[anastasiaguspan]

Suggested change

	1. In the dialog, click **Custom Signal**.
	1. In the drawer, click **Custom Signal**.

[anastasiaguspan]

Suggested change

	- **Scorer name**: The display name for this signal.
	- **Scorer name**: The name for the underlying scorer object.