Trash is a command-line toolkit for generating parsers, parsing input, and analyzing and transforming grammars and parse trees. It supports two workflows: generate interpreter tables and parse directly with Trash, or generate an ANTLR4 parser application for a supported target language.
dotnet trash interp compiles grammar parse trees into .interp files containing
lexer and parser ATNs, together with .tokens vocabularies. These tables can be
loaded by dotnet trash parse without generating or compiling a target-language
parser. Grammar front ends currently support:
- ABNF (
.abnf) — a basic subset, compiled with character-level tokenization and RFC core rules. - ANTLR4 (
.g4) — ANTLR4 grammar syntax. - ANTLR4 Experimental (G4X)
(
.g4x,.g4x) — experimental ANTLR4 grammar syntax for new features, including unconstrained rule names. Rule kind is determined by the grammar declaration rather than capitalization. - Bison
(
.y) — a basic subset, paired with a separate ANTLR4 or G4X lexer grammar. - Invisible XML (iXML) (
.ixml) — a basic subset compiled with character-level tokenization to preserve scannerless matching. - REx
(
.rex) — a basic subset of the EBNF notation used by the REx parser generator. - W3C EBNF (
.ebnf) — a basic subset, compiled with character-level tokenization to preserve scannerless matching.
For example, from the REx example directory:
dotnet trash parse Arithmetic.rex | dotnet trash interp -o interp
dotnet trash parse --allstar -L interp -i '1+2*3' | dotnet trash tree -aSupport for a notation does not imply support for all of its semantics. The ABNF, ANTLR4 Experimental (G4X), Bison, iXML, REx, and W3C EBNF examples document the supported subsets and current limitations. In particular, iXML interpretation produces Trash parse trees, not iXML's prescribed XML output, and the interpreter does not execute arbitrary target-language actions or semantic predicates.
For generated applications, dotnet trash gen supplies ANTLR4 driver and build
templates for supported targets. The grammars-v4 collection
provides grammars for many programming languages and data formats.
Trash also provides XPath 4.0 and XQuery 4.0 expression processing, including XQuery Update operations, to query and edit parse trees. Preserved off-channel tokens, such as whitespace and comments, can participate in source-preserving transformations. Commands compose through shell pipelines using PAX/tar bundles of parse trees, diagnostics, and other artifacts; text-rendering commands can instead output source text, trees, XML, or Graphviz DOT.
Trash runs on Windows, Linux, and macOS as a .NET tool. Invoke its commands as
dotnet trash <command>, for example dotnet trash parse --help.
Interpreted parsing needs no separate target-language toolchain; generating and
building parser applications requires the tools for the selected target.
dotnet tool install -g trash
dotnet tool uninstall -g trash
dotnet new tool-manifest
dotnet tool install trash
Every command is invoked as dotnet trash <name> [options] [args].
Most commands read a parse result set from stdin and write one to stdout,
making them composable with |.
Full invocation rules (argument order, -- separator, MSYS2 quoting) and a
sortable reference table are in
commands.html.
| Command | Description |
|---|---|
| analyze | Analyze a grammar |
| caret | Caret operations on a parse tree |
| clonereplace | Clone and replace in a grammar |
| combine | Combine a split Antlr4 grammar |
| convert | Convert a grammar from one form to another |
| cover | Code coverage analysis |
| dot | Print a parse tree in Graphviz Dot format |
| extract | Extract target-specific code from a grammar |
| ff | Output FIRST and FOLLOW sets of a grammar |
| foldlit | Fold transform on grammar with literals |
| gen | Generate an Antlr4 parser for a given target language |
| genvsc | Generate VS Code extension files |
| glob | Expand glob file patterns |
| iconv | Convert file encoding |
| interp | Generate Antlr4 .interp files from a grammar parse tree |
| itext | Get strings from a PDF file |
| json | Print a parse tree in JSON structured format |
| nullable | Nullable analysis of a grammar |
| parse | Parse a grammar or use a generated parser to parse input |
| perf | Performance analysis of an Antlr grammar parse |
| query | Query parse trees using XPath |
| rename | Rename symbols in a grammar |
| sort | Sort rules in a grammar |
| split | Split a combined Antlr4 grammar |
| sponge | Write parse result set back to files on disk |
| text | Print source text for parse tree nodes |
| tokens | Print tokens in a parse tree |
| tree | Print a parse tree in a human-readable format |
| unfold | Unfold transform on a grammar |
| unfoldlit | Unfold transform with literals on a grammar |
| ungroup | Ungroup transform on a grammar |
| wdog | Kill a program that runs too long |
| xpath | Search parse trees using XPath 4.0 |
| xml | Print a parse tree in XML format |
| xml2 | Enumerate all XPath paths in a parse tree to leaves |
| xquery | Apply XQuery Update expressions to a parse tree |
Runnable examples with step-by-step instructions are in the examples/ directory.
Trash commands pass an ordinary POSIX PAX/tar artifact bundle through
standard input and standard output. Because it is a regular tar stream, a
bundle can also be inspected with standard tools such as tar -tf and
tar -xvf.
Each input parsed by dotnet trash parse normally contributes two regular-file
members:
<name>.ptcontains one JSON-serializedParsingResultSetobject. It holds the parse-tree nodes, parser and lexer metadata, source name, and source text for that input.<name>.errorscontains the parser diagnostics as UTF-8 text. It is empty when the input parsed without errors.
For example, parsing examples/a.g4 and examples/nested/b.g4 can produce:
a.g4.pt
a.g4.errors
nested/b.g4.pt
nested/b.g4.errors
For relative input names, the leading directory supplied to parse is the
automatic bundle root, so ../examples/cmu/a.acmetest becomes
cmu/a.acmetest.pt. Absolute input names use their common directory as the
automatic root. The parse --base-directory DIR option selects the root
explicitly. Artifact names always use / separators and must be relative,
traversal-free paths.
The input's source extension is retained so bundle output has the same names
as generated-target output (for example, a.g4.tree and a.g4.errors).
Parse-tree commands such as xpath, xquery, foldlit, rename, sort,
unfold, unfoldlit, and combine replace the contents of the corresponding
.pt members and pass .errors and other bundle members through unchanged.
Their output is another PAX/tar bundle, so commands remain composable with |.
Commands whose normal output is human-readable text—such as text, tree,
xml, dot, and tokens—still write text by default. Give one of these
commands --bundle to use it as a bundle filter: every .pt member is replaced
by its rendered artifact (for example, .tree), while unrelated members pass
through unchanged.
find examples -type f \
| dotnet trash parse --allstar -L interp -x \
| dotnet trash tree --bundle \
| dotnet trash sponge -o pt-allstar -csponge detects the PAX/tar stream automatically. Under the directory supplied
with -o, it reconstructs every .pt parse tree as source text using its
recorded source filename and copies .errors and other artifacts unchanged. It
rejects absolute paths, . or .. path components, links, duplicate
destinations, and overwrites unless explicitly allowed.
For backward compatibility, commands also auto-detect and read the former JSON array representation of parsing result sets. Newly produced pipeline output is PAX/tar by default.
The source code for the extension is open source, free of charge, and free of ads. For the latest developments on the extension, check out my blog.
git clone https://github.com/kaby76/Trash
cd Trash
make clean; make; make install
You must have the NET SDK version 10 installed to build and run.
See https://github.com/kaby76/Trash/releases.
If you have any questions, email me at ken.domino gmail.com