Support deprecated Gerber syntax via a refactored tokenizer

[nicolube]

Some copper layers exported from ViewMate Pro failed to load — a flood of CoordinateDataWithoutOperationCode errors. Root cause was a mix of newline-based tokenizing and a couple of unsupported deprecated constructs from the Gerber spec. This PR
refactors the tokenizer and adds support for the missing syntax.

Refactored tokenizer (Gerber spec §4.1)

Per §4.1, commands are delimited by the end-of-block character *; newlines are insignificant whitespace. The old tokenizer split input line-by-line, so files that pack the whole stream onto one physical line were parsed as a single command and fell
apart.

Reworked it into a byte-level, */%-aware tokenizer:

• a normal command ends at *
• a %…% extended command is one block, even across multiple lines (e.g. aperture macros)
• outside an extended block a newline still ends the block, so stray junk errors on its own line instead of merging into the next command

Support deprecated single-digit G-codes (Gerber spec §8.3)

§8.3 lists G1/G2/G3 as a tolerated style variation of G01/G02/G03. ViewMate emits these, and we treated them as UnknownCommand — so the interpolation never registered, modal D01 never armed, and every coordinate line in every region errored. Now
supported, including the combined G1X…D1* form, with G3 disambiguated from the G36/G37 region commands.

Richer error context

Errors now carry a column offset and the failing token (Line 14:28: 'G1X1576795Y1969D1*'), not just a line number — which is what made the single-digit G-code issue obvious. Also fixed a path where some errors were silently dropped.

Breaking: GerberParserErrorWithContext.line: Option<(usize, String)> is replaced by context: Option ({ line, offset, token }).

Verification

ViewMate copper layers that previously threw ~597k errors now parse with zero. Regression tests added for each change; full suite, clippy, and fmt are green.

[nicolube]

@hydra Pls review the PR carefully, this is a hevy one with breaking API changes.

[hydra]

Finally got some time to try this, things have been somehwat busy recently and my focus has been elsewhere.

anyway, this PR has the exact same problem as or #26 as per my message: #26 (comment)

[nicolube]

@hydra Hey it seems like u have escapes within ur gcode wich the tokenizer currently not supports:

G04 %MIB1*% *
G04 %IR180*% *
G04 %SFA1.5B3*% *
G04 %OFA60B60*% *
G04 %ASAYBX*% *

I will have a look at it soon...

[nicolube]

@hydra After a quick look into gerber spec section 3.15 (G04) specifies that comment content must be acording to section 3.6.6 wich explicely forbits * and % inside it's content.

So technicly ur gerber is invalid.

[nicolube]

Ahh, I got an old refrence, they have been allowed at a later date...
I will implement it properly according to this spec sheet from this year

https://www.ucamco.com/files/downloads/file_en/554/gerber-layer-format-specification-revision-2026-05_en.pdf?b998de78d4a6b87cd69d06cff46118e5

[nicolube]

Well it is still out of spec....

[hydra]

Ahh yes, this block from demo.gbr

G04 try these commands for image transforms *
G04 %MIB1*% *
G04 %IR180*% *
G04 %SFA1.5B3*% *
G04 %OFA60B60*% *
G04 %ASAYBX*% *

The previous new-line based approach was able handle them as it didn't care about the comment content.

The documentation of the spec isn't particularly clear about escape sequences, but this is what I read today:

Gerber Spec 2026.05, section 3.4.3 Strings

These are expressed by UTF-8 literal characters, or by a Unicode escape. Note that
reserved characters must be escaped.

The Unicode escape for the copyright symbol '©' is as follows:
• With lower case u for a 16-bit hex value: \u00A9
• With upper case U for a 32-bit hex value: \U000000A9

Note: There is no mention of a non-unicode (c-style) escape sequence. e.g. \\ or \* or \%

See also section 3.4.4

The fields follow the string syntax in section 3.4.3 with the additional restriction that a field must
not contain commas. Fields are intended to represent comma-separated items in strings. A
comma can be escaped with \u002C.

field = /[^%*,]*/; # All characters except *%,

I think we need a round-trip test like the ones in reference_file_tests that takes a gerber file with properly escaped comments and fields parses it, then uses gerber_doc_as_str which should escape characters that need to be escaped.

[nicolube]

@hydra

Well also 3.4.3:

Zero-fill if needed to reach the required 4 or 8 hex digits
• The reserved characters '%' (\u0025 ) and '*' (\u002A) must always be escaped as they
are the delimiters of the Gerber syntax.
• '' (\u005C) must be escaped as it is the escape character.
• ',' (\u002C) separates fields, and therefore must be escaped in any string that does not
end the word

Then on next page same section:

Any character may be escaped. Escape every non-ASCII character if you need to keep a file
ASCII-only. This may increase compatibility with legacy software but defeats human readability
of the meta-data.

Sound for me like I could escape them with \...

Want me to keep the escaping (7d1822e) or drop it?

[hydra]

from the changes, there's this snippet from one of the tests:

let reader = gerber_to_reader(
        "
    G04 escaped \\* asterisk and \\% percent survive as comment text*
    %TF.MD5,abc\\%def*%
    M02*
    ",
    );

This doesn't like right to me, given the spec.

I don't see anything in the spec that suggests that escaping * using the sequence \* is allowed. it seems like it should be \u002A for the * character. thus I understand it to be:

G04 escaped \\u002a asterisk and \\u0025 percent survive as comment text*

and

%TF.MD5,abc\\u0025def*%

We need a test for the fields, since there , (comma) also must be escaped.

Or am I missing something in the spec?

[nicolube]

@hydra Shall we parse unicode escapes into unicode chars, or keep as is?