Prevent text encoding corruption

Goal

Garbled text such as — (an em dash after a UTF-8 file was read as Windows-1252 and saved again) can never be committed, no matter which agent, editor or machine made the edit.

Why

Text encoding and AI agents.

Lookup

Mojibake signatures.

House tool: encoding-guard

Dev-Centr ships all of this as one tool in agent-rules tools/encoding-guard/:

  • check-mojibake.mjs: dependency-free Node scanner and repair (--staged, --fix, --json, --allow-non-utf8). It flags a run of characters only when mapping it back to Windows-1252 or IBM437 bytes gives one valid UTF-8 character, so accented text, box drawing and currency signs pass. It also fails files that are not valid UTF-8 at all (a lone ANSI byte written by PowerShell 5.1 Set-Content).

  • install-global-hooks.mjs: sets git config --global core.hooksPath to the tool’s hooks/ folder, keeps any earlier global hooks path chained, and dispatches every common hook type to the repository’s own .git/hooks/<name>.

  • .github/workflows/encoding-guard.yml in agent-rules: the reusable CI workflow (workflow_call).

Rule for agents: general/text-encoding.md in agent-rules. The scripts in the sections below are minimal standalone equivalents for machines and forges where you cannot use the house tool.

Four layers, in order of how much they protect you:

Layer Catches Misses

Machine-wide pre-commit hook

Every commit made on this workstation, in every repository, by any agent or human

Commits made elsewhere; --no-verify; repositories that set their own core.hooksPath

CI check (reusable workflow + org ruleset)

Everything pushed to the forge, including cloud agents and web edits

Nothing reaches history later than this, but a direct push to main is flagged after the fact, not blocked

Agent rule

Lowers how often agents produce mojibake at all

Any agent that did not load the rule, or ignored it

.editorconfig

Editors and formatters writing the wrong charset

Shell one-liners, which ignore it

1. Install the guard machine-wide with core.hooksPath

Hooks live in .git/hooks/, which Git never versions or clones. Committing a hook to a repository protects nobody until each machine installs it, so install the guard once per workstation instead.

core.hooksPath vs init.templateDir

init.templateDir copies hooks into repositories at git init / git clone time only. Existing clones get nothing, and later improvements never reach earlier copies. core.hooksPath points every repository, existing and future, at one directory. That is what you want here. The catch: a global core.hooksPath replaces .git/hooks/, so the guard must chain to the repository’s own hook, as the script below does.

With the house tool, installing is one command, and --status shows what is configured:

node <agent-rules>/tools/encoding-guard/install-global-hooks.mjs
node <agent-rules>/tools/encoding-guard/install-global-hooks.mjs --status --scan <code-root>

Bypass every hook for one commit with git commit --no-verify; skip only the encoding check with ENCODING_GUARD=0 git commit …​. Undo with --uninstall, which restores the previous global hooks path.

Without Node, save this as pre-commit in a hooks directory you own, for example ~/.config/git/hooks/pre-commit. It is ASCII-only on purpose: the signatures are built from octal escapes, so the script cannot be corrupted by the bug it detects.

#!/usr/bin/env bash
# encoding-guard: reject mojibake and UTF-16 in staged text files.
# Installed machine-wide via `git config --global core.hooksPath`.

# 1. Chain to the repository's own pre-commit hook (core.hooksPath hides it).
common_dir=$(git rev-parse --git-common-dir)
if [ -x "$common_dir/hooks/pre-commit" ]; then
  "$common_dir/hooks/pre-commit" "$@" || exit $?
fi

# 2. Signatures: UTF-8 bytes that were read as Windows-1252 and saved again.
sig_lead=$(printf '\303\242\342\202\254')  # "a-circumflex, euro": E2 80 xx (dashes, quotes)
sig_atilde=$(printf '\303\203\302')        # "A-tilde" + Latin-1 char: C3 xx (accented letters)
sig_acirc=$(printf '\303\202\302')         # "A-circumflex" + Latin-1 char: C2 xx (nbsp, degree)

# 3. Intentional examples and fixtures are listed in .encoding-allow
#    (git glob pathspecs, one per line, # comments). .txt is NOT exempt.
excludes=()
if [ -f .encoding-allow ]; then
  while IFS= read -r pat || [ -n "$pat" ]; do
    pat=${pat%
\r'} case "$pat" in ''|'#'*) continue ;; esac excludes+=(":(exclude,glob)$pat") done < .encoding-allow fi files=() while IFS= read -r -d '' f; do files+=("$f"); done < <( git diff --cached --name-only --diff-filter=ACMR -z -- . "${excludes[@]}" ) [ ${#files[@]} -eq 0 ] && exit 0 fail=0 for f in "${files[@]}"; do head=$(git show ":$f" | head -c 4 | od -An -tx1 | tr -d ' \n') case "$head" in fffe*|feff*) echo "encoding-guard: $f is UTF-16/32; save it as UTF-8 without a BOM" >&2; fail=1 ;; efbbbf*) echo "encoding-guard: warning: $f starts with a UTF-8 BOM" >&2 ;; esac done if git grep --cached -nIF -e "$sig_lead" -e "$sig_atilde" -e "$sig_acirc" -- "${files[@]}" >&2; then echo "encoding-guard: mojibake above (UTF-8 read as Windows-1252, then re-saved)." >&2 echo "encoding-guard: repair the file; do not retype or re-save the garbled glyphs." >&2 fail=1 fi exit $fail

Install it (Git Bash, macOS, Linux; on Windows Git runs hooks through its bundled bash, so the same script works):

mkdir -p ~/.config/git/hooks
cp pre-commit ~/.config/git/hooks/pre-commit
chmod +x ~/.config/git/hooks/pre-commit
git config --global core.hooksPath '~/.config/git/hooks'

Verify it fires, in a scratch repository:

git init /tmp/guard-test && cd /tmp/guard-test
printf 'dash: \303\242\342\202\254\342\200\235\n' > bad.md   # an em dash, double-encoded
git add bad.md
git commit -m test    # expected: encoding-guard prints bad.md:1 and the commit is rejected

Notes:

2. Add the CI backstop

The hook only sees commits made on machines that installed it. Cloud agents, web-UI edits, a second laptop and --no-verify all skip it, so run the same check on the forge.

Prefer one reusable workflow for the whole organisation, so the check is maintained in one place:

  1. Put the scan in the organisation’s shared workflow repository (on GitHub, conventionally <org>/.github) as a workflow with on: workflow_call.

  2. Make it required across repositories with an organisation ruleset that requires the workflow to pass, where your plan offers required workflows. New repositories are then covered automatically, with no per-repository file to forget.

  3. Where rulesets are not available, add a two-line caller workflow to each repository, and put it in your template repository so new repositories start with it.

The CI job scans the whole tracked tree, not just the latest diff, so a commit that skipped the hook cannot hide behind a later clean one:

# <org>/.github/.github/workflows/encoding-guard.yml
name: Encoding guard
on:
  workflow_call:
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - name: Reject mojibake and UTF-16 in tracked text
        shell: bash
        run: |
          sig_lead=$(printf '\303\242\342\202\254')
          sig_atilde=$(printf '\303\203\302')
          sig_acirc=$(printf '\303\202\302')
          excludes=()
          if [ -f .encoding-allow ]; then
            while IFS= read -r pat || [ -n "$pat" ]; do
              pat=${pat%\r'}
              case "$pat" in ''|'#'*) continue ;; esac
              excludes+=(":(exclude,glob)$pat")
            done < .encoding-allow
          fi
          fail=0
          while IFS= read -r -d '' f; do
            case "$(head -c 2 "$f" | od -An -tx1 | tr -d ' \n')" in
              fffe|feff) echo "::error file=$f::UTF-16/32 file; save as UTF-8 without a BOM"; fail=1 ;;
            esac
          done < <(git ls-files -z -- . "${excludes[@]}")
          if git grep -nIF -e "$sig_lead" -e "$sig_atilde" -e "$sig_acirc" -- . "${excludes[@]}"; then
            echo "::error::Mojibake found (UTF-8 read as Windows-1252, then re-saved)"
            fail=1
          fi
          exit $fail

Dev-Centr’s own reusable workflow is dev-centr/agent-rules/.github/workflows/encoding-guard.yml (it runs check-mojibake.mjs, so CI and the hook agree). All 15 organisations it is rolled out to were on the GitHub Free plan in October 2026, and so is the owner’s personal account. Organisation rulesets need GitHub Team or higher (on Free, GET /orgs/<org>/rulesets answers 403 "Upgrade to GitHub Team"), so every non-fork, non-archived repository carries its own thin caller workflow instead (the full pattern, including covering new repositories, is in Enforce org-wide checks on GitHub Free):

name: Encoding guard
on:
  push:
    branches: [main]
  pull_request:
jobs:
  encoding:
    uses: dev-centr/agent-rules/.github/workflows/encoding-guard.yml@main

Generic caller, if you are not using a ruleset:

# .github/workflows/encoding-guard.yml in each repository
name: Encoding guard
on: [push, pull_request]
jobs:
  encoding:
    uses: <org>/.github/.github/workflows/encoding-guard.yml@main

A required workflow gates pull requests. If your team pushes straight to main, the check still runs on push and turns the commit red, but it cannot un-push it. That is the reason the machine-wide hook is the primary layer.

Roll it out across an organisation and a personal account

The workflow file lives once; how each repository picks it up depends on who owns the repository.

Organisation-owned repositories

Create an organisation ruleset that targets all repositories (or all with a property), protects the default branch, and requires the encoding-guard workflow from <org>/.github. Repositories created later match the ruleset automatically. Organisation rulesets need GitHub Team or higher; on GitHub Free, fall back to a per-repository caller workflow (below) plus a template repository. Availability of the required-workflow rule also depends on plan and repository visibility, so check the ruleset screen before relying on it.

Personal-account repositories

A user account has no organisation rulesets, so each repository needs the thin caller workflow shown above. It can still call the organisation’s reusable workflow: a reusable workflow in a public repository can be called from any repository, and one in a private repository needs its Actions access setting opened to the callers. Put the caller in your personal template repository so new repositories start with it.

Template repository

Works for both owners, but only for repositories created from it, and only at creation time. It is the fallback, not the mechanism: existing repositories still need the ruleset or a caller file, and a later fix to the template does not reach repositories already created.

Where the logs appear

The workflow code is shared, but every run executes and logs in the repository that triggered it. A failure shows up in that repository’s Actions tab and on that commit’s checks (as encoding / scan for the caller above, or under the ruleset’s required workflow), not in <org>/.github. To see the whole estate, filter the organisation’s Actions or check runs by workflow name rather than looking in the shared repository.

Forks of other people’s projects

Forks are where a sweep does the most damage if you are careless:

  • Fix forward only lines you introduced. Legacy mojibake that came from upstream belongs in an upstream pull request (or an issue), not in a silent rewrite on your fork that will conflict with every future sync.

  • Do not push encoding fixes to a branch that backs an open upstream pull request. The commit lands in the PR immediately and changes what the maintainers are reviewing. Use a new branch and, if the fix is wanted, a separate PR.

  • Do not add your CI workflow to upstream-owned trees. A caller file committed to a fork’s branches travels upstream with the next pull request. In forks, rely on the machine-wide hook; it runs locally and never becomes part of the project.

  • If a fork’s own CI or hooks conflict with yours, the upstream project’s conventions win in its tree.

3. Allowlist intentional mojibake

Some files must contain mojibake on purpose: test fixtures for a repair tool, or a raw sample in a bug report. The house tool offers three opt-outs, narrowest first:

Line marker

Put mojibake-guard: allow anywhere on the line, inside the file’s comment syntax (<!-- mojibake-guard: allow -→ in Markdown or HTML, # mojibake-guard: allow in Python or YAML). Only that line is skipped. AsciiDoc has no end-of-line comment, so use character references there instead (below).

File marker

mojibake-guard: allow-file within the first 10 lines skips the whole file.

.encoding-allow

Globs at the repository root, one per line, # comments. The house tool reads them like .gitignore patterns: a pattern without a slash matches at any depth, dir/ matches everything under it, ** crosses folders. (The standalone scripts above pass the lines to git as :(exclude,glob) pathspecs, which are anchored at the root.)

For example:

# Fixtures for the encoding repair script; these bytes are the test input.
tests/fixtures/mojibake/**

Binaries, lockfiles, .min. and source maps are the only built-in skips.

Prefer the line marker for a single sample line and .encoding-allow for whole fixture files or folders; both are read by check-mojibake.mjs in the hook and in CI alike.

Rules for the allowlist:

  • Name files or fixture folders, never whole file types. *.txt is not exempt: plain-text notes, logs and changelogs get corrupted exactly like .adoc and .md.

  • Add a comment saying why each entry exists.

  • In documentation, prefer not to need an entry: write the garbled sample with character references (AsciiDoc and HTML both accept &#226;&#8364;&#8221;), so the source stays clean and the page still shows the garbage.

4. Give agents the rule

Put this, or your harness’s equivalent, in the always-on agent rules shared by every agent that touches your repositories:

Text files are UTF-8 without a BOM.
- Edit files with the harness's file-edit tool, not shell one-liners.
- Never rewrite text with Windows PowerShell 5.1 Get-Content / Set-Content /
  Out-File / ">" unless the encoding is explicitly UTF-8 without a BOM.
  Prefer pwsh 7, Nushell, or .NET UTF8Encoding($false).
- Never commit with --no-verify to get past the encoding guard.
- If you see garbled text (a-circumflex + euro sign, A-tilde + symbol),
  repair the bytes; never retype or re-save the garbled glyphs.

5. Set .editorconfig

root = true

[*]
charset = utf-8
end_of_line = lf
insert_final_newline = true

charset = utf-8 means UTF-8 without a BOM (utf-8-bom is a separate value). Editors and many formatters honour it. Shell commands do not, which is why this is the last layer, not the first.

6. Edit safely from a shell

When an agent or a script must rewrite a file from the command line:

Nushell

UTF-8 everywhere by default:

open --raw CHANGELOG.adoc | str replace 'old' 'new' | save --force CHANGELOG.adoc
PowerShell 7 (pwsh)

Defaults to UTF-8 without a BOM, so Get-Content -Raw / Set-Content are safe. Check you are really in 7: $PSVersionTable.PSVersion.

Windows PowerShell 5.1 (powershell.exe)

Bypass the cmdlets and use .NET with an explicit no-BOM encoding. .NET resolves relative paths against the process directory, not the PowerShell location, so resolve the path first:

$path = (Resolve-Path .\CHANGELOG.adoc).Path
$utf8 = New-Object System.Text.UTF8Encoding($false)
$text = [System.IO.File]::ReadAllText($path, $utf8)
[System.IO.File]::WriteAllText($path, $text.Replace('old', 'new'), $utf8)

Before capturing output from native tools (git show, node, python), switch the console decoding to UTF-8 so PowerShell does not decode it through the OEM code page:

[Console]::OutputEncoding = New-Object System.Text.UTF8Encoding($false)
$OutputEncoding = [Console]::OutputEncoding

Windows' system-wide "Beta: Use Unicode UTF-8 for worldwide language support" setting makes the ANSI code page UTF-8, which also fixes 5.1’s reads, but it can break older non-Unicode applications. Treat it as a per-machine decision, not a substitute for the guard.

7. Repair what is already in history

  1. Find affected files with the same signatures the guard uses: git grep -nIF -e "$(printf '\303\242\342\202\254')" -e "$(printf '\303\203\302')".

  2. Find the commit that introduced a garbled line: git log -S "$(printf '\303\242\342\202\254')" --oneline — <file>.

  3. Transcode back, do not hand-edit: node <agent-rules>/tools/encoding-guard/check-mojibake.mjs --fix <paths> applies only verified round-trips, unwinds text that was garbled twice (including Windows-1252 on top of IBM437), and re-decodes stray ANSI bytes as Windows-1252. For Antora SVG XML errors on top of that, use the fix-docs-encoding skill in agent-rules.

  4. Characters already replaced by ? or the U+FFFD replacement character cannot be inferred from the bytes. Recover them from the newest earlier revision of the line in git log -p; the tool reports them but never guesses.

  5. Review the diff. A correct repair turns each garbled run back into one character and changes nothing else.

  6. Commit the repair as its own commit. Do not rewrite published history to hide the original; the repair commit is the record.

Related