Skip to content

tooling: make every capability-gated test visible when it is skipped #165

Description

@julia-script

Problem

A local test run can skip an LLVM-dependent assertion without the default reporter showing either
the skip or its reason.

llvmToolchain(...) currently prints a message with console.log and exposes
unavailable(): boolean. Callers return early when it is true. The default reporter does not
reliably print that log, and an early return still counts as a passing test. The run can therefore be
green while the guarded assertion never ran.

The WebAssembly capability helper already has the desired shape: it puts the reason in the skipped
test's name and makes CI fail when a capability promised by the pinned environment is missing. The
LLVM helper has not adopted that behavior, so the repository still has two incompatible
capability-gate patterns.

Current behavior

A local host without opt can report only a passing test:

✓ verifies emitted LLVM modules

Tests  1 passed (1)

Nothing in the default output says that the external opt verification did not run.

Expected behavior

The same run must expose one real skipped test and the reason:

↓ verifies emitted LLVM modules — skipped: opt was not found

Tests  1 skipped (1)

In CI, the missing pinned tool is not an environment-dependent skip. It is a failure explaining
which tool is absent and which assertion could not run.

Required behavior

  • Every capability or external-tool gate uses one shared test-requirement abstraction.
  • A missing local capability produces a skipped test, not an early successful return.
  • The default reporter shows what is missing and what consequently did not run.
  • The summary's skipped count is non-zero.
  • CI fails when a capability or tool promised by the pinned CI environment is absent.
  • No guarded assertion is removed or weakened.

Relevant cases

LLVM tool unavailable locally

const requirement = requireCapability(llvmToolchain(['opt']), 'verifies emitted LLVM modules')
it.skipIf(requirement.skip)(requirement.name, () => verifyWithOpt())

The exact API may differ, but the result must be an observable skip.

Pinned CI tool unavailable

The test must fail before reporting success. The failure must identify opt and the verification
that was lost.

Capability available

The original test name and assertions run normally, with no skipped count.

Evidence

  • test/support/llvmToolchain.ts still logs locally and returns true from unavailable().
  • packages/wasm/test/support/hostCapability.ts already carries the skip reason in the test name
    and converts a missing CI capability into a failure.

Acceptance criteria

  • Running an LLVM-dependent suite without the requested tool shows a reporter-visible skipped
    test and reason.
  • The default summary reports at least one skipped test.
  • Running the same case with CI=1 fails and names the missing tool and guarded assertion.
  • LLVM and WebAssembly capability gates use the same shared mechanism.
  • Capability-present runs execute every original assertion unchanged.
  • Focused tests cover local skip, CI failure, and capability-present execution.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Clear valuebugSomething isn't workingtoolingCLI, formatter, docs tooling

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions