From de627921a69bb94672a0881cfec9864bad8d43ea Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Henrik=20O=2E=20S=C3=B8rensen?= Date: Tue, 4 Aug 2026 14:33:23 +0200 Subject: [PATCH 1/4] feat: Ship the script that removes byte order marks. 2.2 switched SA1412 off, so consumers no longer have to carry a mark on every source file - but nothing removes the ones they already have, and doing it by hand across a tree is where mistakes get made. Stripping the first three bytes blindly destroys a UTF-16 file, whose mark is the only record of its encoding. tools/de-bom.sh reports by default and changes nothing until asked. It refuses to run on a dirty tree, so what it does lands as one commit git checkout can undo. It skips UTF-16, .sln, and the usual build directories, preserves file modes by writing back into the original rather than moving a temp over it, and handles paths with spaces - which matters, because "Service References" has one and an earlier draft reported every file under it as unreadable. POSIX sh with no GNU-isms: od rather than xxd, which ships with vim and is not on every container, and no sed -i, which needs a mandatory argument on BSD and refuses it on GNU. Exercised end to end under a PATH containing only BSD utilities as well as under GNU, against a tree with spaced paths, a UTF-16 file, a mode-640 file and a dirty checkout. The readme says to upgrade before running it. On 2.1.x SA1412 still demands the mark, so stripping first breaks the build on every file. Co-Authored-By: Claude Opus 5 --- README.md | 28 +++++++++++++++ tools/de-bom.sh | 92 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 120 insertions(+) create mode 100755 tools/de-bom.sh diff --git a/README.md b/README.md index 93571e4..4e0b71f 100644 --- a/README.md +++ b/README.md @@ -84,6 +84,34 @@ The encoding check is a build task rather than a diagnostic, so it has its own s false ``` +## Upgrading from 2.1 + +`SA1412`, which required every source file to carry a byte order mark, is off. It never did anything +for the build - a file without a mark compiles fine, since the compiler assumes UTF-8 when none is +present - and what it was quietly protecting against is now `NOTA0001`'s job, which checks the bytes +rather than the mark. + +Nothing forces you to remove the marks you have. If you want to, `tools/de-bom.sh` does it a tree at +a time: + +```sh +tools/de-bom.sh /path/to/repo # report, change nothing +tools/de-bom.sh /path/to/repo --apply # do it +``` + +It reports by default, refuses to run on a dirty tree so the result is one revertible commit, and +leaves UTF-16 files alone - their mark is the only record of the encoding, and removing it destroys +the file. Afterwards, every changed file should differ by exactly one line: + +```sh +git diff --numstat | awk '$1 != 1 || $2 != 1' +``` + +Silence means nothing but marks moved. + +**Take 2.2 first.** On 2.1.x `SA1412` still demands a mark, so stripping them before upgrading breaks +the build on every file. + ## Working on this repository The product here is configuration, and configuration fails silently: a rule that cannot report looks diff --git a/tools/de-bom.sh b/tools/de-bom.sh new file mode 100755 index 0000000..7a10c74 --- /dev/null +++ b/tools/de-bom.sh @@ -0,0 +1,92 @@ +#!/usr/bin/env sh +# +# Removes the UTF-8 byte order mark from source files in a repository. +# +# For trees that carried BOMs because SA1412 demanded one. It does not, and never did, do anything +# for the build: a file without a BOM compiles fine, since the compiler assumes UTF-8 when no mark is +# present. What the mark was quietly protecting against is a file saved in the system codepage, and +# that is now the encoding check's job rather than the BOM's. +# +# Usage: +# de-bom.sh [path] report what would change, touch nothing +# de-bom.sh [path] --apply do it +# +# Reports by default on purpose. This edits every source file in a repository at once, and the first +# thing anyone should see is the list, not the diff. +# +# Refuses to run on a dirty git tree unless --force, so the result is one reviewable commit that +# git checkout can undo. +# +# What it will not touch: +# - UTF-16 files. Their BOM is the only record of the encoding and removing it destroys the file. +# svcutil and EF migrations emit these. +# - .sln files, which some tooling still expects to start with a mark. +# - bin, obj, .git, node_modules, packages. +# +# POSIX sh, no GNU-isms: runs on macOS and in a Linux container alike. Paths with spaces are handled, +# which matters because "Service References" has one. + +set -eu + +root="." +apply=0 +force=0 + +for arg in "$@"; do + case "$arg" in + --apply) apply=1 ;; + --force) force=1 ;; + -h|--help) sed -n '2,30p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; + -*) printf 'unknown option: %s\n' "$arg" >&2; exit 2 ;; + *) root="$arg" ;; + esac +done + +[ -d "$root" ] || { printf 'not a directory: %s\n' "$root" >&2; exit 2; } + +if [ "$apply" -eq 1 ] && [ "$force" -eq 0 ] && git -C "$root" rev-parse --git-dir >/dev/null 2>&1; then + if [ -n "$(git -C "$root" status --porcelain 2>/dev/null)" ]; then + printf 'The tree has uncommitted changes.\n' >&2 + printf 'Commit or stash first, so this lands as one revertible commit - or pass --force.\n' >&2 + exit 1 + fi +fi + +# Extensions worth carrying a BOM historically. Widen if a tree needs it; .sln is left out +# deliberately. +found="$(find "$root" \ + \( -name '*.cs' -o -name '*.csproj' -o -name '*.props' -o -name '*.targets' \ + -o -name '*.json' -o -name '*.resx' -o -name '*.config' -o -name '*.xaml' -o -name '*.md' \) \ + -not -path '*/bin/*' -not -path '*/obj/*' -not -path '*/.git/*' \ + -not -path '*/node_modules/*' -not -path '*/packages/*' \ + -exec sh -c ' + for f do + # Only EF BB BF. FF FE and FE FF are UTF-16 and must keep their mark. + case "$(head -c 3 "$f" | od -An -tx1 | tr -d " \n")" in + efbbbf) printf "%s\n" "$f" ;; + esac + done + ' sh {} +)" + +if [ -z "$found" ]; then + printf 'No UTF-8 byte order marks found under %s\n' "$root" + exit 0 +fi + +count="$(printf '%s\n' "$found" | wc -l | tr -d ' ')" + +if [ "$apply" -eq 0 ]; then + printf '%s\n' "$found" | sed 's|^| |' + printf '\n%s file(s) would have their byte order mark removed.\n' "$count" + printf 'Nothing has been changed. Re-run with --apply.\n' + exit 0 +fi + +printf '%s\n' "$found" | while IFS= read -r f; do + # Write back into the original rather than moving a temp over it, so the inode, the permissions + # and anything watching the file survive. + tail -c +4 "$f" > "$f.debom" && cat "$f.debom" > "$f" && rm -f "$f.debom" +done + +printf '%s file(s) de-BOMed under %s\n' "$count" "$root" +printf 'Review with git diff - every change should be one line, the first one.\n' From 5989be98b5b4e05e28fce61c0dc9dd974a60614f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Henrik=20O=2E=20S=C3=B8rensen?= Date: Tue, 4 Aug 2026 15:41:38 +0200 Subject: [PATCH 2/4] fix: Take the byte order mark off this repository's own csproj. It predates everything here and nothing enforces marks in either direction now, so it was harmless - but a repository shipping a script to remove them should not be carrying one. Removed with that script, which is also the first time it has been run against something other than a test tree: one file, one line, and the build, all six rules and the packaging check unchanged afterwards. The upgrade notes gain the second half of the ordering problem. Take 2.2 before stripping marks, which was already there - and then close the IDE while stripping them, which was not. Visual Studio and Rider fix a file's encoding when they open it and keep it for the buffer, so a file opened with a mark has one written back on the next save regardless of what is now on disk. An editor left running undoes the script quietly, file by file, as you touch them. Co-Authored-By: Claude Opus 5 --- Nota.CodeAnalysis/Nota.CodeAnalysis.csproj | 2 +- README.md | 8 ++++++++ 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/Nota.CodeAnalysis/Nota.CodeAnalysis.csproj b/Nota.CodeAnalysis/Nota.CodeAnalysis.csproj index dc2c206..15a18d0 100644 --- a/Nota.CodeAnalysis/Nota.CodeAnalysis.csproj +++ b/Nota.CodeAnalysis/Nota.CodeAnalysis.csproj @@ -1,4 +1,4 @@ - + netstandard2.0 true diff --git a/README.md b/README.md index 4e0b71f..53040a5 100644 --- a/README.md +++ b/README.md @@ -109,9 +109,17 @@ git diff --numstat | awk '$1 != 1 || $2 != 1' Silence means nothing but marks moved. +Two things in that order, and both bite if you get them wrong. + **Take 2.2 first.** On 2.1.x `SA1412` still demands a mark, so stripping them before upgrading breaks the build on every file. +**Then close the IDE while you strip them.** Visual Studio and Rider decide a file's encoding when +they open it and keep that decision for the buffer. A file that was opened with a mark gets one +written back on the next save, whatever the file on disk now looks like - so an editor left running +quietly undoes the script, file by file, as you touch them. Closing it and reopening afterwards is +enough; the encoding is re-detected from what is actually there. + ## Working on this repository The product here is configuration, and configuration fails silently: a rule that cannot report looks From 5d2177cab436cd8e3fcccefcc8daeace3098ef57 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Henrik=20O=2E=20S=C3=B8rensen?= Date: Tue, 4 Aug 2026 16:01:57 +0200 Subject: [PATCH 3/4] fix: Treat Notalib as first party as well as Nota. The other team's code has been under Notalib for years, and one root does not cover the other: a prefix only matches at a dot boundary, so Notalib.Something is not under "Nota" - the same rule that stops "System" swallowing "SystemsManager". Without both, half of Nota's own code sorted among the vendors. This is also the case that cannot be inferred. A file in Nota.Something has no way to know Notalib is ours too, which is exactly why the setting exists alongside the inference. Verified against the built globalconfig: with both roots, Nota.* and Notalib.* both land in the last block. Worth knowing, since separate_roots is on: they are two roots, so they get a run each rather than sharing one. Nota.* and Notalib.* run together is reported as UA1001. That is the setting working as asked rather than a surprise, but it is a visible change in every file that imports both. Co-Authored-By: Claude Opus 5 --- .../content/Nota.CodeAnalysis.globalconfig | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/Nota.CodeAnalysis/content/Nota.CodeAnalysis.globalconfig b/Nota.CodeAnalysis/content/Nota.CodeAnalysis.globalconfig index 3e82801..e604344 100644 --- a/Nota.CodeAnalysis/content/Nota.CodeAnalysis.globalconfig +++ b/Nota.CodeAnalysis/content/Nota.CodeAnalysis.globalconfig @@ -23,9 +23,11 @@ dotnet_separate_import_directive_groups = true # and dotnet_separate_import_directive_groups groups by first-level namespace with no notion of whose # code it is. # -# Defaulted to Nota, because that is what nearly every consumer's own code is called, and a setting -# every repository has to remember is a setting most repositories will not have. A consumer whose -# code is called something else overrides it in their own .editorconfig: +# Defaulted to the two roots Nota's own code uses, because a setting every repository has to remember +# is a setting most repositories will not have. Both are needed: a root only matches at a dot +# boundary, so Notalib.Something is not covered by "Nota" - the same rule that stops "System" +# swallowing "SystemsManager". A consumer whose code is called something else overrides it in their +# own .editorconfig: # # [*.cs] # usinglayout.first_party_prefixes = Contoso, Fabrikam @@ -36,7 +38,7 @@ dotnet_separate_import_directive_groups = true # # A repository that gets this wrong still gets a sensible layout - its own namespaces are simply # treated as one more vendor - which is why it is worth defaulting rather than demanding. -usinglayout.first_party_prefixes = Nota +usinglayout.first_party_prefixes = Nota, Notalib usinglayout.separate_roots = true ## naming rules From b8de0ee8c7dfb3a0bdb4fcb22b6c0228383231c0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Henrik=20O=2E=20S=C3=B8rensen?= Date: Tue, 4 Aug 2026 16:41:24 +0200 Subject: [PATCH 4/4] feat: Take UsingLayoutAnalyser 0.3.0, and call it 2.2.1. 0.3.0 makes first_party_prefixes optional: with nothing configured, each file is judged against the namespace it declares. That does not change anything here, because this package sets the prefixes explicitly and configuration still wins - which is the point of taking it. A consumer who has never set them stops having their own code sorted among their vendors, quietly, for want of a line in .editorconfig. It also raises the floor on which SDKs the using rules load at all: 0.3.0 inherits 0.2.1's build against Roslyn 4.8, so anything from the .NET 8 SDK upwards runs them rather than skipping them with CS9057. Nota and Notalib still have to be named. A file in Nota.Something has no way to know Notalib is ours too, which is exactly the case inference cannot reach. Verified rather than assumed: builds clean, all six rules report, the encoding check passes, the packaging check passes, and the packed nuspec declares UsingLayoutAnalyser 0.3.0. Co-Authored-By: Claude Opus 5 --- .../Nota.CodeAnalysis.Verification.csproj | 2 +- Nota.CodeAnalysis/Nota.CodeAnalysis.csproj | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Nota.CodeAnalysis.Verification/Nota.CodeAnalysis.Verification.csproj b/Nota.CodeAnalysis.Verification/Nota.CodeAnalysis.Verification.csproj index e316aaa..69286f4 100644 --- a/Nota.CodeAnalysis.Verification/Nota.CodeAnalysis.Verification.csproj +++ b/Nota.CodeAnalysis.Verification/Nota.CodeAnalysis.Verification.csproj @@ -51,7 +51,7 @@ - + diff --git a/Nota.CodeAnalysis/Nota.CodeAnalysis.csproj b/Nota.CodeAnalysis/Nota.CodeAnalysis.csproj index 15a18d0..492b3f0 100644 --- a/Nota.CodeAnalysis/Nota.CodeAnalysis.csproj +++ b/Nota.CodeAnalysis/Nota.CodeAnalysis.csproj @@ -6,7 +6,7 @@ true - 2.2.0 + 2.2.1 content/README.md @@ -24,7 +24,7 @@ - +