A shared vocabulary that every format writes in its own dialect

.github/workflows/ci.yml

@@ -23,7 +23,7 @@ jobs:
       # SDK (the MSBuild build host crashes). Whitespace rules are syntactic, so nothing is lost.
       - name: Verify formatting
         run: |
-          for dir in MarkupString MarkupString.Ansi MarkupString.Html MarkupString.Tests MarkupString.AotSmoke; do
+          for dir in MarkupString MarkupString.Ansi MarkupString.Html MarkupString.Mxp MarkupString.Pueblo MarkupString.Tests MarkupString.AotSmoke; do
             dotnet format whitespace --folder "$dir" --exclude "**/bin/**" --exclude "**/obj/**" --verify-no-changes
           done
 
@@ -70,7 +70,7 @@ jobs:
         with:
           dotnet-version: 10.0.x
 
-      # The packages promise IsAotCompatible. Publishing the smoke app with all three assemblies
+      # The packages promise IsAotCompatible. Publishing the smoke app with every package assembly
       # rooted analyses every public entry point, so a trim- or AOT-hostile addition fails here
       # instead of in a consumer's app. Any IL2xxx/IL3xxx warning is an error.
       - name: Publish natively and fail on any trim/AOT warning

CHANGELOG.md

@@ -1,7 +1,7 @@
 # Changelog
 
-All notable changes to `MarkupString`, `MarkupString.Ansi` and `MarkupString.Html`. The three
-packages share one version and are released together.
+All notable changes to `MarkupString`, `MarkupString.Ansi`, `MarkupString.Html`, `MarkupString.Mxp`
+and `MarkupString.Pueblo`. The packages share one version and are released together.
 
 The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project
 follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
@@ -10,13 +10,64 @@ follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ### Added
 
+- **A shared vocabulary.** A game puts a thing in its text once and every format writes it in its own
+  dialect, or stands something else in for it. The types are core's and say what a thing is; each
+  format's package says how it is written:
+
+  | Factory | MXP | Pueblo | HTML | ANSI / Plain / BBCode |
+  |---|---|---|---|---|
+  | `MarkupText.Sound`, `Music` | `<SOUND>`, `<MUSIC>` | `<img xch_sound="play"\|"loop">` | `<audio>` | nothing |
+  | `MarkupText.StopSound` | `<SOUND Off>`, `<MUSIC Off>` | `<img xch_sound="stop">` | `ms-sound-stop` | nothing |
+  | `MarkupText.Image` | `<IMAGE>` | `<img>` | `<img>` | its description, or its address; BBCode `[img]` |
+  | `MarkupText.Pane` | `<FRAME>` + `<DEST>` | `<xch_pane action="redirect">` and back | `ms-pane` | its text |
+  | `MarkupText.ClearScreen` | nothing | `<xch_page clear="text">` | `ms-clear` | ANSI `ESC[H ESC[2J` |
+  | `MarkupText.Prefetch` | nothing | `<xch_prefetch>` | `<link rel="prefetch">` | nothing |
+  | `MarkupText.ExpireLinks` | `<EXPIRE>` | nothing | `ms-expire` | nothing |
+  | `MarkupText.Variable`, `Gauge`, `Status` | `<VAR>`, `<GAUGE>`, `<STAT>` | their text | `ms-variable`, `ms-gauge`, `ms-status` | their text |
+  | `MarkupText.Relocate`, `LoginPrompt` | `<RELOCATE>`, `<USER>`, `<PASSWORD>` | nothing | nothing | nothing |
+
+  A link is already this kind of thing (`AnsiMarkup` with a `LinkKind`), so a picture inside a link is
+  a picture that is a link in every dialect. The vocabulary serialises as built-in kinds, so storing it
+  needs no package registered.
+- **Points.** `IPointMarkup` is a markup that stands at a point rather than marking text — a sound, a
+  bell, a clear. It rides on its own `IPointMarkup.Carrier` (a zero-width space, `MarkupText.PointCarrier`;
+  the bell keeps its U+0007), and `MarkupText.Point(markup)` builds one of your own. A format with no
+  emitter for a point writes nothing at all — not the carrier, and not the layers around it. A point
+  marks its carrier and nothing else: `MarkupText.Wrap` refuses one over other text, and a cover read
+  back with a point out of place drops the point and keeps the text.
+- **`MarkupString.Mxp`**, a new package. `WithMxp()` writes the vocabulary as MXP's secure elements.
+  `WithMxp(supports)` holds each element to what the client answered to `<SUPPORT>`: one it refused is
+  written as a format without MXP writes it — nothing for a sound, the description for a picture, the
+  text in the main window for a pane. `MxpRegistration.Elements` lists the names to ask about.
+- **`MarkupString.Pueblo`**, a new package. `WithPueblo()` writes the vocabulary in the Pueblo
+  client's own extensions, with the names and attributes its source reads.
 - **A bell.** `MarkupText.Bell()` marks a point in the text where the client is asked to get someone's
   attention: U+0007 for a terminal, Pueblo or MXP client, and an empty `<span class="ms-bell"
   role="alert">` for HTML, where the page decides what a bell means. Plain and BBCode leave nothing
-  behind. It rides on the single U+0007 it marks, which measures zero display cells, so it survives
-  slicing, concatenation and padding as a point in the string without moving anything laid out around
-  it — and it is the only way to get a control character into rendered output, since the encodings
-  drop them from ordinary text.
+  behind. It is a point riding on the U+0007 it marks, which measures zero display cells, so it survives
+  slicing, concatenation and padding without moving anything laid out around it — and it is the only
+  way to get a control character into rendered output, since the encodings drop them from ordinary
+  text.
+
+### Changed
+
+- **`ToPlainText()`, `ToString()` and equality leave out point carriers.** A sound or a bell is not
+  text a reader or a pattern sees: `Concat(Sound(...), Plain("hi"))` equals `Plain("hi")`. `Text`
+  still holds the carriers, so positions, slicing and padding are unchanged.
+- **The kinds core serialises itself are reserved.** `MarkupRegistry.With(IMarkupCodec)` refuses a
+  codec claiming `"neutral"` or one of the shared vocabulary's kinds, which are written and read
+  without consulting a registry; a codec registered under one would write text that read back as
+  something else.
+- **A layer from another package nested inside ANSI styling now renders inside it.** The Ansi
+  package's set emitters used to wrap every layer they do not own around their own output, whatever
+  the nesting: `Wrap(red, Wrap(HtmlMarkup "b", "x"))` gave `<b><span style="color: …">x</span></b>`.
+  It now gives `<span style="color: …"><b>x</b></span>`, and a picture inside a command link stays
+  inside the link rather than replacing it. A layer *between* two styled layers keeps its place too:
+  `[bold, tag, red]` is a bold inside a tag inside a red, rather than one folded bold-red inside a tag.
+  A terminal is unchanged — a style there is state rather than nesting, so the sequence is still
+  written once around the run.
+
+## 2.2.0 — 2026-09-19
 
 ### Added
 
@@ -148,5 +199,5 @@ follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
   for existing SGR text, a state-diffing `SgrWriter`, and emitters for all six formats.
 - `MarkupString.Html` — `HtmlMarkup` for raw HTML/MXP tags, folding `b`/`i`/`u`/`s` into terminal
   styling through `IAnsiStyleSource`, and `HtmlCss.Fixed` for the `ms-*` classes the emitters write.
-- All three packages are `IsAotCompatible`, with a native-AOT publish in CI that fails on any
+- Every package is `IsAotCompatible`, with a native-AOT publish in CI that fails on any
   trim or AOT warning.

Directory.Build.props

@@ -11,7 +11,7 @@
   </PropertyGroup>
 
   <!--
-    Package metadata that is identical across the three shipping packages. Everything that
+    Package metadata that is identical across the shipping packages. Everything that
     differs per package — PackageId, AssemblyName, Description, project references — stays in
     the project file. Test and smoke projects set IsPackable=false, so these are inert there.
   -->

MarkupString.Ansi/AnsiRegistration.cs

@@ -5,7 +5,9 @@ public static class AnsiRegistration
 {
 	/// <summary>
 	/// Returns a registry that renders <see cref="AnsiMarkup"/> in all six formats and serialises
-	/// it under kind <c>"ansi"</c>. Plain needs nothing: the body passes through.
+	/// it under kind <c>"ansi"</c>. Plain needs nothing: the body passes through. It also writes the
+	/// bell, a <see cref="ClearScreenMarkup"/> for a terminal, and an <see cref="ImageMarkup"/> as
+	/// BBCode's <c>[img]</c>.
 	/// </summary>
 	/// <remarks>
 	/// The five emitters are <see cref="IMarkupSetEmitter"/>s — one per format, claiming the whole
@@ -30,6 +32,9 @@ public static MarkupRegistry WithAnsi(this MarkupRegistry registry)
 			.With(new BellEmitter(MarkupFormat.Ansi))
 			.With(new BellEmitter(MarkupFormat.Pueblo))
 			.With(new BellEmitter(MarkupFormat.Mxp))
-			.With(new BellEmitter(MarkupFormat.Html));
+			.With(new BellEmitter(MarkupFormat.Html))
+			// From the shared vocabulary, what a terminal and BBCode can express.
+			.With(new ClearScreenAnsiEmitter())
+			.With(new ImageBBCodeEmitter());
 	}
 }

MarkupString.Ansi/Emitters/AnsiBBCodeEmitter.cs

@@ -17,8 +17,13 @@ public bool TryEmit(MarkupSet set, ReadOnlySpan<char> body, in EmitContext conte
 		ArgumentNullException.ThrowIfNull(set);
 		ArgumentNullException.ThrowIfNull(output);
 
-		var style = AnsiEmitterSupport.Fold(set, context.Format);
+		AnsiEmitterSupport.EmitSegmented(set, body, context, output, WriteTags);
+		return true;
+	}
 
+	/// <summary>Writes one stretch of folded layers as BBCode tags.</summary>
+	private static void WriteTags(in AnsiStyle style, ReadOnlySpan<char> body, in EmitContext context, IBufferWriter<char> output)
+	{
 		// Reverse video has no BBCode form either, so the colour that would show as the text colour
 		// is the one written.
 		var foreground = style.Inverted ? style.Background : style.Foreground;
@@ -30,35 +35,30 @@ public bool TryEmit(MarkupSet set, ReadOnlySpan<char> body, in EmitContext conte
 			? candidate
 			: null;
 
-		using var core = new PooledCharWriter(body.Length + 64);
-
 		if (hex.Length > 0)
 		{
-			core.Write("[color=");
-			core.Write(hex);
-			core.Write("]");
+			output.Write("[color=");
+			output.Write(hex);
+			output.Write("]");
 		}
 		if (link is not null)
 		{
-			core.Write("[url=");
-			core.Write(link);
-			core.Write("]");
+			output.Write("[url=");
+			output.Write(link);
+			output.Write("]");
 		}
-		if (style.Bold) core.Write("[b]");
-		if (style.Italic) core.Write("[i]");
-		if (style.Underlined) core.Write("[u]");
-		if (style.StrikeThrough) core.Write("[s]");
-
-		core.Write(body);
+		if (style.Bold) output.Write("[b]");
+		if (style.Italic) output.Write("[i]");
+		if (style.Underlined) output.Write("[u]");
+		if (style.StrikeThrough) output.Write("[s]");
 
-		if (style.StrikeThrough) core.Write("[/s]");
-		if (style.Underlined) core.Write("[/u]");
-		if (style.Italic) core.Write("[/i]");
-		if (style.Bold) core.Write("[/b]");
-		if (link is not null) core.Write("[/url]");
-		if (hex.Length > 0) core.Write("[/color]");
+		output.Write(body);
 
-		AnsiEmitterSupport.WriteWrapped(set, core.WrittenSpan, context, output);
-		return true;
+		if (style.StrikeThrough) output.Write("[/s]");
+		if (style.Underlined) output.Write("[/u]");
+		if (style.Italic) output.Write("[/i]");
+		if (style.Bold) output.Write("[/b]");
+		if (link is not null) output.Write("[/url]");
+		if (hex.Length > 0) output.Write("[/color]");
 	}
 }

MarkupString.Ansi/Emitters/AnsiEmitterSupport.cs

@@ -52,11 +52,107 @@ internal static AnsiStyle Fold(MarkupSet? set, MarkupFormat format)
 		return effective;
 	}
 
+	/// <summary>
+	/// Writes one contiguous stretch of layers this package folds, around a body the layers inside it
+	/// have already been written into.
+	/// </summary>
+	internal delegate void SegmentWriter(in AnsiStyle style, ReadOnlySpan<char> body, in EmitContext context, IBufferWriter<char> output);
+
+	/// <summary>
+	/// Writes a run in the nesting it was built with, for a format that expresses nesting: the layers
+	/// this package folds are written by <paramref name="writeSegment"/> in stretches, and a layer it
+	/// does not own is written by its own emitter between them. <c>[bold, tag, red]</c> is a bold inside
+	/// a tag inside a red, and comes out that way, rather than as one bold red inside a tag.
+	/// </summary>
+	/// <remarks>
+	/// The common shapes cost nothing extra: with no foreign layer carrying an emitter, this is one
+	/// <paramref name="writeSegment"/> over the whole fold, the same call the emitters made before there
+	/// was anything to interleave.
+	/// </remarks>
+	internal static void EmitSegmented(
+		MarkupSet set,
+		ReadOnlySpan<char> body,
+		in EmitContext context,
+		IBufferWriter<char> output,
+		SegmentWriter writeSegment)
+	{
+		if (!HasDelegatedLayer(set, context))
+		{
+			writeSegment(Fold(set, context.Format), body, context, output);
+			return;
+		}
+
+		PooledCharWriter? front = null;
+		PooledCharWriter? back = null;
+		try
+		{
+			front = new PooledCharWriter(body.Length + 32);
+			front.Write(body);
+			back = new PooledCharWriter(front.WrittenCount + 32);
+
+			var pending = AnsiStyle.None;
+			var hasPending = false;
+
+			// A style that clears discards everything around it (AnsiStyle.Combine), and a delegated layer
+			// in between does not change that: once a stretch has cleared, the stretches outside it write
+			// no styling at all.
+			var cleared = false;
+			for (var i = 0; i < set.Count; i++)
+			{
+				if (ClaimsStyle(set[i], context.Format, out var style))
+				{
+					// Innermost first, and an inner layer's settings win, which is how Fold combines them.
+					pending = hasPending ? style.Combine(pending) : style;
+					hasPending = true;
+					continue;
+				}
+
+				var emitter = context.Registry.FindEmitter(set[i].GetType(), context.Format);
+				if (emitter is null) continue;
+
+				if (hasPending)
+				{
+					cleared |= pending.Clear;
+					back.Clear();
+					writeSegment(cleared ? AnsiStyle.None : pending, front.WrittenSpan, context, back);
+					(front, back) = (back, front);
+					pending = AnsiStyle.None;
+					hasPending = false;
+				}
+
+				back.Clear();
+				emitter.Emit(set[i], front.WrittenSpan, context, back);
+				(front, back) = (back, front);
+			}
+
+			writeSegment(cleared ? AnsiStyle.None : pending, front.WrittenSpan, context, output);
+		}
+		finally
+		{
+			front?.Dispose();
+			back?.Dispose();
+		}
+	}
+
+	/// <summary>Whether any layer is one this package does not fold and something else can write.</summary>
+	private static bool HasDelegatedLayer(MarkupSet set, in EmitContext context)
+	{
+		for (var i = 0; i < set.Count; i++)
+			if (!ClaimsStyle(set[i], context.Format, out _)
+				&& context.Registry.FindEmitter(set[i].GetType(), context.Format) is not null)
+				return true;
+
+		return false;
+	}
+
 	/// <summary>
 	/// Writes <paramref name="core"/> — the run as this package rendered it — wrapped by the layers
-	/// this package does not own in <see cref="EmitContext.Format"/>, innermost first, each through
-	/// its own emitter for that format. A layer with no emitter registered for the format wraps in
-	/// nothing: its body passes through.
+	/// this package does not own in <see cref="EmitContext.Format"/>, innermost first, each through its
+	/// own emitter for that format. A layer with no emitter registered for the format wraps in nothing:
+	/// its body passes through. This is the terminal's shape, where a style is state rather than
+	/// nesting: the sequence is written once around the run, and where a delegated layer's own output
+	/// sits relative to it changes nothing on screen. Formats that express nesting use
+	/// <see cref="EmitSegmented"/>.
 	/// </summary>
 	internal static void WriteWrapped(
 		MarkupSet set,
@@ -132,14 +228,15 @@ internal static void EmitTagged(
 		IBufferWriter<char> output,
 		TagFlavour flavour)
 	{
-		var style = Fold(set, context.Format);
-
-		using var core = new PooledCharWriter(body.Length + 64);
-		SgrWriter.Transition(AnsiStyle.None, style, core);
-		WriteTaggedLink(style, body, core, flavour);
-		if (LeavesState(style)) SgrWriter.Reset(core);
-
-		WriteWrapped(set, core.WrittenSpan, context, output);
+		// The flavour cannot ride on the delegate's signature, so it is closed over here; a run with
+		// nothing delegated never allocates the closure's work beyond this one call.
+		EmitSegmented(set, body, context, output,
+			(in AnsiStyle style, ReadOnlySpan<char> segment, in EmitContext segmentContext, IBufferWriter<char> segmentOutput) =>
+			{
+				SgrWriter.Transition(AnsiStyle.None, style, segmentOutput);
+				WriteTaggedLink(style, segment, segmentOutput, flavour);
+				if (LeavesState(style)) SgrWriter.Reset(segmentOutput);
+			});
 	}
 
 	/// <summary>