Skip to content

feat: Add Markdown plain text projection - #12

Merged
Fuyeors merged 1 commit into
mainfrom
feat/plain-text-projection
Aug 28, 2026
Merged

feat: Add Markdown plain text projection#12
Fuyeors merged 1 commit into
mainfrom
feat/plain-text-projection

Conversation

@Fuyeors

@Fuyeors Fuyeors commented Aug 28, 2026

Copy link
Copy Markdown
Member

Summary

This PR ports the TypeScript FFM Markdown parser to Rust and provides a search-oriented plain-text projection from the parsed AST. The projection is independent from Tantivy: the parser exposes text, while the search backend owns schema, tokenizer, field boosting, storage, and snippets.

to_plain_text()

The Rust crate now exports:

pub fn to_plain_text(nodes: &[AstNode]) -> String

The implementation performs one direct AST traversal. It keeps visible text, heading/paragraph/list/table/blockquote content, inline and fenced code, link labels, hard breaks, and FFM item titles/bodies. It omits Markdown presentation markers, link destinations, CSS class/style metadata, and structural thematic-break markers.

HTML and CSS are deliberately not parsed or sanitized by this API. When the parser represents them as text, they remain ordinary text and are returned unchanged. A canonical regression case covers a CSS <style> element. The function does not render HTML, strip tags, or depend on Tantivy.

Shared fixtures

fixtures/markdown.json and fixtures/ffm.json now include plain_text expectations for the standard Markdown and FFM cases. The shared cases cover headings, inline/fenced code, links, raw HTML/CSS text, tables, lists, hard breaks, FFM quote/accordion/chain/slide nodes, Unicode domains, and nesting behavior. TypeScript exposes a matching toPlainText reference implementation, and both language runners consume the same expected values.

Validation

  • pnpm --filter @fuyeor/markdown-parser typecheck — passed
  • pnpm --filter @fuyeor/markdown-parser test — 38 passed
  • pnpm --filter @fuyeor/linkify test — 20 passed
  • cargo fmt --all -- --check — passed
  • cargo check --workspace --all-targets --all-features — passed
  • cargo clippy --workspace --all-targets --all-features -- -D warnings — passed
  • cargo test --workspace --all-targets --all-features — 2 thin runners passed

The existing CommonMark/GFM compatibility harness remains separate because it has its own filtered corpus and skip rules. The canonical cross-language suite is the source of truth for Rust/TypeScript plain-text parity.

@Fuyeors
Fuyeors merged commit 2b1ccb8 into main Aug 28, 2026
2 checks passed
@Fuyeors
Fuyeors deleted the feat/plain-text-projection branch August 30, 2026 20:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant