Skip to content

feat(ctrace): support formatted multi-route ITM/DWT trace - #2609

Open
thorstendb-ARM wants to merge 33 commits into
mainfrom
dev-ctrace-decode-tree
Open

thorstendb-ARM wants to merge 33 commits into
mainfrom
dev-ctrace-decode-tree

Conversation

@thorstendb-ARM

@thorstendb-ARM thorstendb-ARM commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Fixes

  • No linked issue.

Changes

  • Normalize ctrace-run metadata into explicit trace routes, with processor bindings where resolvable, while accepting optional YAML null values and deferring backend-specific requirements until output preflight.
  • With an explicit trace format, select exactly one eligible SWO, TB, or named-TB raw input and decode either an unformatted single-route stream or complete 16-byte memory-aligned CoreSight formatter frames through one OpenCSD DecodeTree.
  • Preserve undeclared legacy raw-SWO input while diagnosing but otherwise ignoring unrelated TB and Event Recorder inputs.
  • Bind one ITM decode pipeline, including DWT post-decoding, and all timestamp, recovery, diagnostic, and pending-packet state to each configured Trace Bus route.
  • Preserve legacy unformatted SWO decoding and CSV/CTF layout, honor stream filters for its public stream 0, and write architectural CoreSight Trace Bus IDs to formatted CSV rows.
  • Generate one lazy CTF stream per emitted formatted route, with explicit route clock domains, processor metadata, deterministic cleanup, and data-driven Trace Compass views.
  • Centralize CTF event-type and topic definitions and verify their completeness at compile time.
  • Keep valid multi-clock CTF bundles while omitting the companion XML with one warning when Trace Compass cannot establish a common timeline.
  • Add real and synthetic formatted-trace fixtures, fixture-integrity validation, external Babeltrace coverage, focused unit/integration coverage, and expanded supported-platform CI.
  • Update the architecture, constraints, CTF profile, implementation decision record, test documentation, and architecture SVG.

Scope

  • This PR preserves undeclared legacy raw-SWO input and supports explicitly selected unformatted single-route or formatted multi-route ITM/DWT input from one eligible SWO, TB, or named-TB raw file. Formatted input requires complete 16-byte memory-aligned CoreSight frames.
  • The root-level trace-format field is a documented private working contract until format and framing metadata are standardized in CMSIS-Toolbox.
  • ETM/ETE/PTM, MTB, Event Recorder, FSYNC/HSYNC framing, multiple simultaneous raw inputs, and cross-clock correlation remain separate follow-up work.
  • No external submodule or devtools-global workflow behavior is changed.

Validation

  • Debug and Release: all 7 registered ctrace CTests passed.
  • Unit tests: 347 total; all passed on Linux, with 342 passed and 5 platform-specific special-file tests skipped on macOS.
  • Integration tests: 30/30 passed.
  • Coverage: 100% ctrace source-line coverage; branch coverage 93.6%.
  • Babeltrace 2.0.5 external-consumer gate passed.
  • Real formatted TB fixture: 4096 raw bytes decoded into 1055 semantic events; CSV contains 213 rows for stream 1 and 312 rows for stream 2; CTF contains separate stream_1 and stream_2 files.
  • Raw SWO capture: 19999 bytes decoded into 8599 semantic events and successfully converted to CSV, CTF stream_0, and Trace Compass XML.
  • Regression coverage confirms that raw SWO without IDs remains supported and that excluding public stream 0 requires no timestamp clock and emits neither a CTF stream nor Trace Compass XML.
  • Trace Compass Server 0.17.0 / TSP 0.6.0 acceptance covered single-clock route views, processor labels, conditional topics, exceptions, and valid multi-clock CTF without XML.

Checklist

  • 🤖 This change is covered by unit tests (if applicable).
  • 🤹 Manual testing has been performed (if necessary).
  • 🛡️ Security impacts have been considered (if relevant).
  • 📖 Documentation updates are complete (if required).
  • 🧠 Third-party dependencies and TPIP updated (if required).

@github-actions

github-actions Bot commented Sep 14, 2026

Copy link
Copy Markdown

Test Results

    6 files   -    37      6 suites   - 121   5s ⏱️ - 19m 11s
  382 tests  -   767    382 ✅  -   750   0 💤  - 17  0 ❌ ±0 
1 146 runs   - 1 291  1 132 ✅  - 1 247  14 💤  - 44  0 ❌ ±0 

Results for commit 214263e. ± Comparison against base commit da65548.

This pull request removes 953 and adds 186 tests. Note that renamed tests count towards both.
AuxCmdTests ‑ MkdirCmdTest
AuxCmdTests ‑ RmdirCmdTest
AuxCmdTests ‑ TouchCmdTest
BuildSystemGeneratorTests ‑ GenAuditFile
BuildSystemGeneratorTests ‑ GenAuditFile_WithOut_Existing_Audit_File
BuildSystemGeneratorTests ‑ GenAuditFile_With_Existing_Audit_File
BuildSystemGeneratorTests ‑ GetString
BuildSystemGeneratorTests ‑ StrConv
BuildSystemGeneratorTests ‑ StrNorm
CBuildGCCTests ‑ Asm
…
CtraceIntegTests ‑ AbortsAllOutputsOnFormattedDataBeforeFirstSourceId
CtraceIntegTests ‑ AppliesMultiValueTypeAndStreamUnionsAndTheirIntersection
CtraceIntegTests ‑ CompletesHealthyBackendWhenOtherOutputTargetHasWrongType
CtraceIntegTests ‑ ConvertsBlinkyFixtureToGoldenOutputsAndSkipsLegacyExcludedTraceBusInput
CtraceIntegTests ‑ ConvertsReconstructedFormattedTraceBusFixture
CtraceIntegTests ‑ DecodesDeterministicSyntheticFormattedPacketFamiliesOnAnchorAndFallbackRoutes
CtraceIntegTests ‑ DecodesExplicitUnformattedNamedTraceBuffer
CtraceIntegTests ‑ DefersAbsentAndNullFormattedClocksToCtfOutputValidation
CtraceIntegTests ‑ ExcludesUnformattedRawSwoWithoutRequiringClock
CtraceIntegTests ‑ PublishesOutputsWithUnresolvedFormattedRouteRecovery
…

♻️ This comment has been updated with latest results.

Comment thread tools/ctrace/src/output/ctf/TraceCompassXmlWriter.cpp Fixed
Comment thread tools/ctrace/src/output/ctf/TraceCompassXmlWriter.cpp Fixed
Comment thread tools/ctrace/src/tracerun/CtraceRunMeta.cpp Fixed
Comment thread tools/ctrace/src/tracerun/CtraceRunMeta.cpp Fixed
Comment thread tools/ctrace/src/tracerun/CtraceRunMeta.cpp Fixed
Comment thread tools/ctrace/src/tracerun/CtraceRunMeta.cpp Fixed
Comment thread tools/ctrace/src/tracerun/CtraceRunMeta.cpp Fixed
Comment thread tools/ctrace/src/tracerun/CtraceRunMeta.cpp Fixed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The legacy fallback can emit stream_0 even when a stream filter excludes it.

Pull request overview

Adds formatted, multi-route ITM/DWT trace support while preserving legacy SWO behavior.

Changes:

  • Adds normalized routes and formatted CoreSight decoding.
  • Adds route-aware CSV, CTF, and Trace Compass output.
  • Expands fixtures, tests, CI validation, and documentation.
File summaries
File Reviewed change
tools/ctrace/test/unit/support/TestSupport.h Shared route-aware test helpers.
tools/ctrace/test/unit/support/OpenCsdTestSupport.h OpenCSD route test support.
tools/ctrace/test/unit/support/CtfTestSupport.h CTF parsing test helpers.
tools/ctrace/test/unit/src/output/TraceOutputLifecycleTests.cpp Output lifecycle coverage.
tools/ctrace/test/unit/src/output/ctf/CtfStreamWriterTests.cpp Route-labeled stream tests.
tools/ctrace/test/unit/src/output/ctf/CtfSchemaTests.cpp CTF schema coverage.
tools/ctrace/test/unit/src/output/csv/CsvRowMapperTests.cpp Route serialization tests.
tools/ctrace/test/unit/src/output/csv/CsvFileOutputTests.cpp Route-aware CSV tests.
tools/ctrace/test/unit/src/model/TraceSelectionTests.cpp Route and stream selection tests.
tools/ctrace/test/unit/src/diagnostics/DiagnosticsTests.cpp Per-route diagnostics tests.
tools/ctrace/test/unit/src/decode/OpenCsdErrorControllerTests.cpp Structured error handling tests.
tools/ctrace/test/unit/src/decode/DwtPacketDecoderTests.cpp Route-aware DWT tests.
tools/ctrace/test/unit/src/decode/CortexMPostDecoderTests.cpp Route-local decoder state tests.
tools/ctrace/test/unit/src/control/DecodeConsumersTests.cpp Route-aware consumer tests.
tools/ctrace/test/unit/CMakeLists.txt Registers unit tests.
tools/ctrace/test/integration/src/ValidateFixtureIntegrity.cmake Validates fixture integrity.
tools/ctrace/test/integration/src/ValidateBabeltrace2Consumer.cmake Validates Babeltrace consumption.
tools/ctrace/test/integration/README.md Documents integration validation.
tools/ctrace/test/integration/CMakeLists.txt Registers integration tests.
tools/ctrace/test/data/TB-Trace/split_tb_trace.py Splits TB fixture data.
tools/ctrace/test/data/TB-Trace/README.md Documents TB fixtures.
tools/ctrace/test/data/TB-Trace/Blinky+Arm.ctrace-run.yml Defines formatted fixture metadata.
tools/ctrace/test/data/README.md Documents trace fixtures.
tools/ctrace/test/data/formatted-synthetic/Synthetic.ctrace-run.yml Defines synthetic formatted routes.
tools/ctrace/test/data/formatted-synthetic/README.md Documents synthetic fixtures.
tools/ctrace/test/data/Blinky+Arm/expected/Blinky+Arm.SWO.traceanalysis.xml Updates legacy expected XML.
tools/ctrace/test/data/.gitattributes Configures fixture handling.
tools/ctrace/src/tracerun/TraceRunDiscovery.h Declares input discovery.
tools/ctrace/src/tracerun/TraceRunDiscovery.cpp Selects eligible raw inputs.
tools/ctrace/src/tracerun/TraceRunConfig.h Stores trace-format configuration.
tools/ctrace/src/tracerun/CtraceRunMeta.h Declares normalized metadata.
tools/ctrace/src/output/TraceOutputConfig.h Stores output configuration.
tools/ctrace/src/output/ctf/TraceCompassXmlWriter.h Declares Trace Compass XML output.
tools/ctrace/src/output/ctf/CtfUuid.h Declares CTF UUID utilities.
tools/ctrace/src/output/ctf/CtfUuid.cpp Implements CTF UUID handling.
tools/ctrace/src/output/ctf/CtfStreamWriter.h Declares stream layout options.
tools/ctrace/src/output/ctf/CtfStreamWriter.cpp Writes route-labeled streams.
tools/ctrace/src/output/ctf/CtfMetadataWriter.h Declares metadata writing.
tools/ctrace/src/output/ctf/CtfMetadataModel.h Declares CTF topology.
tools/ctrace/src/output/ctf/CtfGraphicalTopic.h Defines graphical topics.
tools/ctrace/src/output/ctf/CtfExceptionLaneTracker.h Declares exception lane tracking.
tools/ctrace/src/output/ctf/CtfExceptionLaneTracker.cpp Tracks route-local exception lanes.
tools/ctrace/src/output/ctf/CtfEncoder.h Declares CTF encoding.
tools/ctrace/src/output/ctf/CtfBundleOutput.h Declares CTF bundle output.
tools/ctrace/src/output/ctf/CtfBundleOutput.cpp Coordinates CTF bundles and XML.
tools/ctrace/src/output/csv/CsvRowMapper.cpp Serializes route IDs.
tools/ctrace/src/model/TraceSelection.h Declares route-aware selection.
tools/ctrace/src/model/TraceSelection.cpp Applies route-aware selection.
tools/ctrace/src/model/TraceRoute.h Defines normalized routes.
tools/ctrace/src/model/TraceEvent.h Adds route identity to events.
tools/ctrace/src/model/CoreSightFormatter.h Defines formatter contracts.
tools/ctrace/src/diagnostics/TraceIssueReporter.h Declares diagnostic reporting.
tools/ctrace/src/diagnostics/TraceIssueReporter.cpp Reports per-route diagnostics.
tools/ctrace/src/decode/OpenCsdTreeSession.h Declares DecodeTree ownership.
tools/ctrace/src/decode/OpenCsdTraceElement.h Defines normalized trace elements.
tools/ctrace/src/decode/OpenCsdItmSession.h Declares unformatted ITM sessions.
tools/ctrace/src/decode/OpenCsdItmSession.cpp Preserves unformatted sessions.
tools/ctrace/src/decode/OpenCsdItmDecoder.h Declares ITM decoding.
tools/ctrace/src/decode/OpenCsdFormattedItmSession.h Declares formatted sessions.
tools/ctrace/src/decode/OpenCsdErrorController.h Declares error recovery control.
tools/ctrace/src/decode/OpenCsdErrorController.cpp Collects structured errors.
tools/ctrace/src/decode/DwtPacketDecoder.h Declares route-aware DWT decoding.
tools/ctrace/src/decode/DwtPacketDecoder.cpp Carries route identity through DWT decoding.
tools/ctrace/src/decode/DecodePipeline.h Declares decode pipelines.
tools/ctrace/src/decode/DecodePipeline.cpp Supports formatted pipelines.
tools/ctrace/src/decode/CortexMStreamDecoder.h Declares route-aware stream state.
tools/ctrace/src/decode/CortexMStreamDecoder.cpp Implements per-route stream decoding.
tools/ctrace/src/decode/CortexMPostDecoder.h Declares post-decoder interfaces.
tools/ctrace/src/decode/CortexMPostDecoder.cpp Maintains route-local semantic state.
tools/ctrace/src/control/TraceDirectoryJob.cpp Integrates normalized input selection.
tools/ctrace/src/control/FileDecodeJob.h Declares decode-job interfaces.
tools/ctrace/src/control/DecodeConsumers.h Declares route-aware consumers.
tools/ctrace/src/control/DecodeConsumers.cpp Forwards route-aware events.
tools/ctrace/src/CMakeLists.txt Registers implementation files.
tools/ctrace/README.md Documents formatted trace support.
tools/ctrace/docs/todo.md Updates deferred scope.
tools/ctrace/docs/multicore-multisource-plan.md Records routing design.
tools/ctrace/docs/constraints.md Documents implementation constraints.
tools/ctrace/CMakeLists.txt Updates build configuration.
.github/workflows/ctrace.yml Runs expanded CI validation.
.github/matrix_includes_ctrace.json Expands supported CI platforms.
Review details

Suppressed comments (1)

tools/ctrace/src/output/OutputRequirements.cpp:174

  • When a legacy unformatted run is invoked with a stream filter that excludes public stream 0 (for example, --stream 1), selectedCtfRoutes is empty, but this fallback unconditionally re-adds the synthetic route. CTF preflight then requires timestamps.clock and the legacy encoder can create stream_0 even though no configured route is selected, contradicting the documented no-emitted-route behavior. Only apply the compatibility fallback when the selection includes the synthetic route.
  • Files reviewed: 102/119 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@codecov

codecov Bot commented Sep 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.22%. Comparing base (da65548) to head (214263e).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2609      +/-   ##
==========================================
+ Coverage   72.64%   74.22%   +1.57%     
==========================================
  Files         185      197      +12     
  Lines       30676    32515    +1839     
  Branches    18058    19016     +958     
==========================================
+ Hits        22286    24135    +1849     
+ Misses       6080     6072       -8     
+ Partials     2310     2308       -2     
Flag Coverage Δ
ctrace-cov 100.00% <100.00%> (+0.20%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
tools/ctrace/src/control/DecodeConsumers.cpp 100.00% <100.00%> (ø)
tools/ctrace/src/control/FileDecodeJob.cpp 100.00% <100.00%> (ø)
tools/ctrace/src/control/TraceDirectoryJob.cpp 100.00% <100.00%> (ø)
tools/ctrace/src/decode/CortexMPostDecoder.cpp 100.00% <100.00%> (ø)
tools/ctrace/src/decode/CortexMPostDecoder.h 100.00% <100.00%> (ø)
tools/ctrace/src/decode/CortexMStreamDecoder.cpp 100.00% <100.00%> (ø)
tools/ctrace/src/decode/DecodePipeline.cpp 100.00% <100.00%> (ø)
tools/ctrace/src/decode/DecodePipeline.h 100.00% <ø> (ø)
tools/ctrace/src/decode/DwtPacketDecoder.cpp 100.00% <100.00%> (ø)
tools/ctrace/src/decode/OpenCsdErrorController.cpp 100.00% <100.00%> (ø)
... and 37 more

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants