Skip to content

SKILL.md

Latest commit

 

History

History
116 lines (92 loc) · 5.09 KB

SKILL.md

File metadata and controls

116 lines (92 loc) · 5.09 KB
Raw
name playwright-e2e
description Hybrid E2E UI testing workflow for AI coding agents (Qwen Code, Claude Code, Cursor). Use Browser MCP / Playwright MCP to EXPLORE the live app, write a Markdown test plan, and GENERATE or DEBUG Playwright Test specs — then EXECUTE the suite through the Playwright Test runner as a queue (sequential, with retries) producing HTML + JUnit artifacts and a Markdown summary. Includes a CI/CD template for GitHub Actions with a self-hosted macOS runner. Use when the user wants to create, run, or maintain interface/UI automated tests, needs queue execution with rerun-of-failed, wants Markdown artifacts per run, or wants to wire E2E into CI/CD. Do NOT use MCP as the test executor — MCP is the authoring/debugging layer only.

Playwright E2E — Hybrid Agent Workflow

Core principle (the hybrid)

Browser MCP (Playwright MCP) and Playwright Test are two different layers, not interchangeable:

Layer Role Traits
Playwright MCP Authoring & debugging Interactive, explores live app, writes plan + specs, investigates failures. Token-heavy, non-deterministic.
Playwright Test Execution Queue, retries, JUnit/HTML artifacts, CI gate. Deterministic, cheap, headless.

NEVER run the suite "through MCP" as the executor. Generate/repair specs with MCP, then execute with npx playwright test. MCP cannot give CI a pass/fail gate, a reproducible suite, or detached runs.

When to use this skill

  • User wants to create UI/E2E automated tests for a web app.
  • User wants tests run as a queue (sequential, with retries), leaving Markdown artifacts.
  • User wants a CI/CD setup (GitHub Actions, self-hosted Mac).
  • User is driving the work with an AI agent + Playwright MCP.

Workflow

Phase 1 — Generate (MCP)

  1. Connect Playwright MCP (npx @playwright/mcp).
  2. Explore the target flow live; write a human-readable plan to specs/<flow>.md (steps as a numbered list — this is the "artifact" the user asked for).
  3. Generate tests/<flow>.spec.ts following references/best-practices.md.
  4. If a spec already exists but is flaky/broken, use MCP to open the app and inspect the real DOM/locators before editing.

Phase 2 — Execute (runner)

Run the queue via the bundled script:

bash scripts/run_queue.sh --rerun-failed
# or a subset:
bash scripts/run_queue.sh --grep "checkout"

The script runs npx playwright test with list + html + junit reporters, optionally reruns only failed specs once, and converts JUnit → Markdown.

Phase 3 — Artifacts (Markdown)

scripts/junit_to_md.py turns test-results/junit.xml into reports/run-<timestamp>.md with a pass/fail table, durations, and failure details + a verdict line (✅ ALL GREEN / ❌ HAS FAILURES). This is the per-run Markdown artifact.

Phase 4 — CI/CD

Drop templates/github-actions-mac.yml into .github/workflows/. It uses a self-hosted macos runner (for Safari/WebKit fidelity), installs deps, runs the queue with --rerun-failed, and uploads both playwright-report/ (HTML+trace) and reports/ (Markdown) as artifacts.

Scripts

  • scripts/run_queue.sh — queue execution + rerun-failed + artifact conversion.
  • scripts/junit_to_md.py — JUnit XML → Markdown report.

Templates

  • templates/playwright.config.ts — reporters (list/html/junit), retries, trace: 'on-first-retry', screenshot/video on failure, single worker in CI.
  • templates/github-actions-mac.yml — GitHub Actions self-hosted Mac workflow.

Anti-patterns (full list in references/best-practices.md)

  1. No hard sleeps (waitForTimeout(2000)) — use web-first assertions / auto-wait.
  2. No document.querySelector inside evaluate() — use locators.
  3. No chained tests (test 2 depends on test 1) — each test self-contained.
  4. Prefer getByRole / getByTestId over CSS/XPath.
  5. Auth via fixtures / storageState, not re-login in every test.
  6. Assert on behavior, not implementation; use expect(...).toBeVisible().

Pitfalls

  • MCP-as-executor: no suite, no gate, token blow-up. Keep MCP for authoring.
  • Flaky by design: fixed sleeps and chained tests produce false reds.
  • Missing junit output: configure the junit reporter to a file in playwright.config.ts, not just --reporter=junit (which prints to stdout).
  • Self-hosted Mac runner not registered: the workflow waits forever — verify gh runner is online before relying on CI.

Verification

  • Local: bash scripts/run_queue.sh --rerun-failed → expect reports/run-*.md with a verdict line.
  • CI: push a branch, open the Actions run, confirm both artifacts uploaded.
  • Quality gate: fail the pipeline when junit_to_md.py verdict is ❌ HAS FAILURES (exit code from npx playwright test already does this).

Portability to Qwen Code

This skill uses the Agent Skills standard. To use it in Qwen Code:

# from the skill directory
cp -r . ~/.qwen/skills/playwright-e2e/   # or your Qwen skills path

SKILL.md format, scripts, and templates are tool-agnostic (bash + python3 + node). No Hermes-specific APIs are used.