# BINARY — TODO archive (2026-05)

Archived entries from `BINARY_TODO.md` per LLMP-DEC retention policy. IDs preserved (never reassigned). Format identical to active file.

## ACCORE-BIN-T-N9G6: Add non-generic `Type`-based `Serialize(object, Type, ...)` overloads
**Priority:** P2 · **Type:** Feature · **Status:** Closed (2026-05-04) · **Related:** `ACCORE-BIN-T-T8K3`

### Resolution

Added in `AcBinarySerializer.cs`:
- `Serialize(object?, Type, opts)` → `byte[]`
- `Serialize(object?, Type, IBufferWriter<byte>, opts)` → `int`
- `SerializeChunked(object?, Type, PipeWriter, opts)` → `int`
- `SerializeChunkedFramed(object?, Type, PipeWriter, opts)` → `int`

`AcBinaryDeserializer.cs` already had `Deserialize(byte[], Type, opts)` / `Deserialize(ReadOnlySequence<byte>, Type, opts)` / `Deserialize(AsyncPipeReaderInput, Type, opts)` overloads — no new entries needed.

**Layering note**: `PipeReader → AsyncPipeReaderInput` drain-loop is the consumer's responsibility, not the binary serializer's. The serializer surface ends at `AsyncPipeReaderInput`; transport-specific draining (PipeReader, NamedPipe, SignalR `state.Buffer.Write`, etc.) lives in the consumer layer (e.g. `AcBinaryInputFormatter`, `AcBinaryHubProtocol.TryParseChunkData`).

Consumed by ASP.NET Core MVC formatter package (`AyCode.Services/Mvc/`) — `AcBinaryInputFormatter`, `AcBinaryOutputFormatter`, `AddAcBinaryFormatters` extension. Media type: `application/vnd.acbinary`. Drain-loop inlined in `AcBinaryInputFormatter.ReadRequestBodyAsync`.

Plugin frameworks, ASP.NET ModelBinding, DI middleware, and DataContractSerializer-style "generic-API container" use-cases need to serialize an `object` whose type is known only at runtime. Current AcBinary surface forces a reflection trampoline through the generic `Serialize<T>`:

```csharp
// Today's workaround (slow + noisy):
typeof(AcBinarySerializer).GetMethod("Serialize", new[] { type, typeof(AcBinarySerializerOptions) })
    .MakeGenericMethod(type).Invoke(null, new[] { value, options });
```

**Implementation outline:**
- `public static byte[] Serialize(object? value, Type type, AcBinarySerializerOptions? options = null)`
- `public static int Serialize(object? value, Type type, IBufferWriter<byte> writer, AcBinarySerializerOptions? options = null)`
- `public static int SerializeChunked(object? value, Type type, PipeWriter writer, AcBinarySerializerOptions? options = null)` and `Pipe` overload
- `public static int SerializeChunkedFramed(object? value, Type type, PipeWriter writer, AcBinarySerializerOptions? options = null)` and `Pipe` overload
- `public static ValueTask SerializeAsync(object? value, Type type, Stream stream, ...)` — coordinated with `ACCORE-BIN-T-T8K3`
- Internal dispatch: `value.GetType()` is the runtime type; the `Type type` parameter constrains the **declared** type for polymorphism handling (`ObjectWithTypeName` write decision).

**Acceptance:**
- All non-generic overloads round-trip via the generic deserializer's `Deserialize(byte[], Type)` overload.
- Plugin-style scenario: serialize `IList<dynamic>` of mixed-type elements → all elements correctly typed in the wire output.
- API doc-strings call out the performance characteristics (slightly slower than generic due to runtime `Type` lookup but **without** the reflection trampoline cost).

## ACCORE-BIN-T-H2Q6: Fixed-width dual-length string header (Small/Medium/Big) for 1-pass decode
**Priority:** P1 · **Type:** Wire-format + Performance · **Status:** Closed (2026-05-06) · **Related:** `DecodeUtf8SinglePass`, `CountUtf8Chars`, `WriteStringWithDispatch`, `ReadStringUtf8`

Current Compact string decode uses two-pass flow for non-ASCII payloads (`CountUtf8Chars` + `DecodeUtf8SinglePass`).
Planned direction: remove VarUInt-based string-length path for the new string wire variant, and carry both lengths in a fixed-width header so deserialize can allocate target `string` immediately and decode in a single pass.

### Planned format tiers

- **Small**: packed `uint16` (`charLen:8 | utf8Len:8`)
- **Medium**: packed `uint32` (`charLen:16 | utf8Len:16`)
- **Big**: `uint32 charLen + uint32 utf8Len`

Writer picks the smallest fitting tier; reader dispatches by marker and reads fixed-width lengths (no VarUInt loop for string length metadata).

### Why

- Removes `CountUtf8Chars` pass on the new markers (1-pass decode path)
- Keeps decode branch profile stable (fixed-size header reads)
- Maintains range safety with explicit Big overflow path

### Constraints captured from current benchmark context

- Performance evaluation target is non-ASCII-heavy data (ASCII-shortcuts intentionally not primary)
- Wire-format backward compatibility is not required for this development phase

### Marker layout decision (2026-05-06)

After analysis on the new "all UTF-8 Magyar" benchmark baseline (`2026-05-06_13-10-30.LLM` — Compact +5-25% slower than MemPack on every cell):

**Confirmed**: the previous benchmark's Compact-vs-MemPack advantage was an artifact of ASCII property names hitting the `FixStrAscii` / Latin1-widen fast path; once string property values are also UTF-8 Magyar, the actual hot path (`EncodeUtf8SinglePass` + two-pass `CountUtf8Chars` + `DecodeUtf8SinglePass`) becomes the bottleneck.

**Marker scope decision** — clean split between ASCII fast path and non-ASCII tier dispatch:

**MEGMARAD (changeless)**:
- `FixStrAscii` (≤31 byte ASCII) — kompakt 1-byte header + Latin1 widen, zero UTF-8 decode pipeline
- `StringAscii` (>31 byte ASCII) — long ASCII fast path, Latin1 widen
- `StringInternRef` — 2nd+ occurrence of interned string (no body, just cache index — not affected by 2-pass problem)
- `StringEmpty`, `Null` — sentinel markers

**MEGSZŰNIK (replaced by H2Q6 tiers)**:
- `FixStr` (32 marker values 103-134 — non-ASCII short) → replaced by `StringSmall`
- `String` (1 marker value 91 — non-ASCII long with VarUInt utf8Len) → replaced by `StringSmall` / `StringMedium` / `StringBig`
- `StringInternFirst` (1 marker value 94 — VarUInt utf8Len interning) → replaced by `StringInternFirstSmall` / `StringInternFirstMedium`

**ÚJ markers** (5 total):
- `StringSmall` — non-ASCII, `[marker:1][charLen:8][utf8Len:8][bytes]`, utf8Len ≤ 255
- `StringMedium` — non-ASCII, `[marker:1][charLen:16][utf8Len:16][bytes]`, utf8Len ≤ 65535
- `StringBig` — non-ASCII, `[marker:1][charLen:32][utf8Len:32][bytes]`, utf8Len > 65535
- `StringInternFirstSmall` — `[marker:1][cacheIdx:VarUInt][charLen:8][utf8Len:8][bytes]`
- `StringInternFirstMedium` — `[marker:1][cacheIdx:VarUInt][charLen:16][utf8Len:16][bytes]`

**Trade-off justification**:
- Wire cost on short non-ASCII strings: +2 byte/string header (3 vs 1) → ~0.07-0.36% wire growth on Repeated cell (10 short Magyar string × 2 byte / 28 KB)
- CPU saving: `CountUtf8Chars` Pass 1 eliminated on every non-ASCII string decode → directly attacks the +25% Deser baseline gap
- The 2-byte hybrid `FixStr` (non-ASCII) variant (1 byte marker + 1 byte charLen) was considered but **rejected**: marginal wire saving (-1 byte vs StringSmall) does not justify the +1 marker complexity given the tiny absolute wire impact on the Repeated cell. Cleaner to have ASCII-vs-non-ASCII at the marker level (FixStrAscii vs StringSmall/Medium/Big).

**Interning tier sizing rationale**:
- `MaxStringInternLength` is `byte`-typed (`AcBinarySerializerOptions.cs:125`, default 64, abszolút max 255 char)
- Worst-case: 255 char × 4 byte/char (emoji-only) = 1020 byte → fits in Medium tier (utf8Len ≤ 65535)
- Realistic Magyar/CJK: 64 char × 2-3 byte = 128-192 byte → Small tier
- **Big tier never engages on the interning path** — only Small + Medium needed (+2 markers, not +3)

### Marker address space reservation (post-H2Q6)

The marker reorg frees **34 marker values** (32 `FixStr` non-ASCII + `String` + `StringInternFirst`). After allocating 5 for H2Q6, **29 values remain free**. Strategic reservation plan to prevent ad-hoc consumption and minimize future wire-format breaks:

| Reserved range | Count | Future feature | Status |
|---|---|---|---|
| `StringSmall` / `StringMedium` / `StringBig` | 3 | H2Q6 Compact tiers | **active (this entry)** |
| `StringInternFirstSmall` / `StringInternFirstMedium` | 2 | H2Q6 interning tiers | **active (this entry)** |
| `FixArrayBase..FixArrayMax` | 16 | `ACCORE-BIN-T-L9Y3` (FixArray short-list count in marker) | reserved, future |
| Sentinel-length string tier markers | ~5 | `ACCORE-BIN-T-S5L8` (sentinel-length encoding) | reserved, future |
| Markerless schema lane | ~4 | `ACCORE-BIN-T-S2X9` (markerless schema lane opt-in) | reserved, future |
| `StringFastWire` | 1 | `ACCORE-BIN-T-F3W6` (dedicated FastWire string marker) | reserved, future |
| General reserve | 3 | unallocated | tartalék |

**Wire-format version bump**: v2 → v3 at H2Q6 landing. The reserved-but-unimplemented marker values are documented but not yet decoded — readers throw `unknown marker` if wire contains them. Future activation of `FixArray` / sentinel-length / markerless schema lane within the **same v3 wire format** is non-breaking for already-deployed v3 consumers (they reject unknown markers cleanly; producers opt in to emit them).

### Acceptance

- New string markers implemented for Small/Medium/Big tiers + InternFirstSmall/InternFirstMedium tiers
- Deserialize path for these markers performs single-pass decode without `CountUtf8Chars`
- 29 freed marker values strategically reserved per the address-space reservation table; documented in `BinaryTypeCode.cs` with `// Reserved for ACCORE-BIN-T-XXXX (future)` comments
- Wire-format version bump v2 → v3 documented in `BINARY_FORMAT.md`
- Existing round-trip tests pass, plus new boundary tests for tier transitions (utf8Len = 254/255/256/65534/65535/65536) and interning tier transitions
- Benchmark report includes before/after for Compact mode on non-ASCII dataset (Ser/Deser/RT + Size) vs the `2026-05-06_13-10-30.LLM` baseline

### Resolution

Landed 2026-05-06. End-to-end implementation: marker reorg + writer tier-dispatch + reader tier-readers + SGen template + skip path + interning path. Five new markers (`StringSmall`/`Medium`/`Big`/`InternFirstSmall`/`InternFirstMedium`) replacing the old `String`/`StringInternFirst`/`FixStrBase..Max` (32 + 1 + 1 = 34 marker values freed, 5 used; 29 reserved for future features per the address-space plan). Wire format version bumped v2 → v3.

Follow-up A-direction header pack-write/read optimization landed in the same window: `Unsafe.WriteUnaligned<ushort>` (Small) / `<uint>` (Medium) / `<ulong>` (Big) replace 2× byte / 2× ushort / 2× uint stores; reader uses single `uint`/`ulong` loads with bit-extract. Direct `ref byte` writes (no Span-shape overhead).

**Tests:** 222 pass / 13 pre-existing GuidIId failures (unchanged). 55/55 Utf8TranscoderTests pass.

**Benchmark vs `2026-05-06_13-10-30.LLM` baseline (`2026-05-07_08-55-49.LLM`, immediately post-H2Q6):**
- Compact-vs-MemPack Deser ratio improvement on baseline gap: **-14 to -28 percentage points** across cells
- Deser: **4/5 cells now faster than MemPack** (Small -6%, Medium -3%, Large -9%, Deep -7%); Repeated cell remaining +5% gap (V4N2 Phase 3 SIMD multi-byte transcoder targets this)
- Wire size: **5/5 cells smaller than MemPack** (-8% to -11%)
- Ser: 1/5 win (Large -9%), 1/5 tie (Medium 0%), 3/5 minor lag (+2-7% Small/Repeated/Deep) — host-noise band

**Bench evolution post-H2Q6** (subsequent micro-opts on the same H2Q6 base):
- `2026-05-07_09-39-09.LLM` — A irány header pack-write/read (`Unsafe.WriteUnaligned` ushort/uint/ulong): zaj-szintű mozgás, strukturális javulás
- `2026-05-07_15-13-39.LLM` — V4N4 Step 1+2 method-split (`AggressiveInlining`): **regresszió** (Small Ser +29.6 pp, Repeated Ser +8.9 pp) → `WriteStringSmallFast` túl-aggresszív inline-olás code-bloat / i-cache pressure
- `2026-05-07_15-29-21.LLM` — V4N4 finomított (NoInlining a SmallFast-ra, dispatcher hint nélkül, Reader split visszavonva): **konszolidált state**:
  - **Ser**: 5/5 cell paritás-vagy-jobb (Small **-8.5%**, Medium ≈, Large **-8.5%**, Repeated ≈, Deep ≈)
  - **Deser**: **4/5 cell faster than MemPack** (Medium -4.7%, Large **-10.6%**, Repeated **-3.8%**, Deep **-10.1%**); Small +10% remaining gap
  - **Wire**: 5/5 cell -8% to -11% smaller (unchanged)
  - **Net**: Compact mostantól 8/10 cellán nyer Compact vs MemPack; csak Small Deser-en marad +10% gap (kis abszolút érték, ~1 µs)

**Critical algorithmic correctness lesson** (from V4N3 follow-up `GetUtf8ByteCount`): the initial 4-popcount formula assumed `lowSur == highSur` per chunk. Fix: 5-popcount closed-form. Caught by surrogate-pair-split-across-chunk regression tests. Documented in Utf8Transcoder.

**Marker address space (post-H2Q6, v3 wire):**
- 91 → StringSmall (was String)
- 94 → StringMedium (was StringInternFirst)
- 103 → StringBig
- 104 → StringInternFirstSmall
- 105 → StringInternFirstMedium
- 106..134 reserved (29 values: 16 for `L9Y3` FixArray, 5 for `S5L8` sentinel-length, 4 for `S2X9` markerless schema lane, 1 for `F3W6` FastWire dedicated marker, 3 reserve)

**Related follow-up TODO entries (now Open):** `O7G2` (overflow guard), `S6F2` (shift-mentes Small fast path), `W2C8` (WASM string-cache H2Q6 maximalizálás).

## ACCORE-BIN-T-M3R7: ASCII marker-dispatch — writer detect + reader dedicated path
**Priority:** P2 · **Type:** Performance + wire optimization · **Related:** `BinaryTypeCode.FixStrAsciiBase..StringAscii` markers, `WriteStringWithDispatch`, `ReadAsciiBytesAsString`
**Status:** Closed (2026-05-04)

> **Sorrendi megjegyzés:** ezt **AZ ENCODER OPTIMALIZÁCIÓ UTÁN** csináljuk (lásd `ACCORE-BIN-T-E2F9`). Indok: a custom encoder/decoder Vector256 ASCII narrow/widen path-jai már magukban gyorsan kezelik az ASCII byte-ot. A marker-dispatch ezen FELÜL csak a per-call dispatch-overhead spórolást hozza (no `Ascii.IsValid` scan, no decoder layer). Garantált win, de additív — méréstechnikailag tisztább a decoder/encoder utánra hagyni.

The `FixStrAscii*` (135-166) and `StringAscii` (167) markers are defined in `BinaryTypeCode.cs` with helper methods (`IsAsciiString`, `IsFixStrAscii`, `EncodeFixStrAscii`, `DecodeFixStrAsciiLength`). Encoding/decoding logic NOT yet implemented — currently both writer and reader use the universal `String` / `FixStr` markers.

### Implementation
- **Writer**: in `WriteStringUtf8` / `WriteFixStrDirect`, after UTF-8 encoding (D-2 path), check `bytesWritten == charLength` (= ASCII iff equal). If ASCII, emit `FixStrAscii` (≤31 byte) or `StringAscii` (>31 byte). Else emit existing `FixStr` / `String`. Free detect — both numbers already computed by D-2.
- **Reader**: in `ReadStringUtf8` (or upstream marker dispatch), branch on marker. ASCII markers → dedicated byte→char widening path (no UTF-8 decode, no `Ascii.IsValid` scan, no decoder dispatch). Non-ASCII markers → existing custom UTF-8 decoder.
- **SGen**: regenerate readers/writers to dispatch on the new markers.
- **Re-enable ASCII fast paths**: uncomment writer FixStr dispatch in `AcBinarySerializer.cs` and reader `Ascii.IsValid` block in `ReadStringUtf8` — these temporarily disabled blocks become the marker-aware paths (no IsValid scan needed since the marker is the contract).

### Wire format change
- Format version bump (1 → 2). Old readers fail clean on new wire (version mismatch). New readers must reject old wire OR support backward read.

### Acceptance
- Repeated Strings (Hungarian content) Deser: AcBinary closes the ~10% gap vs MemoryPack
- Pure ASCII tests (Small/Medium/Large/Deep): AcBinary Ser AND Deser ≥ MemoryPack
- Wire size: minimum -25% vs MemoryPack across all test cells
- SGen-generated code compiles and round-trips on all `[AcBinarySerializable]` types
- Decision documented: backward-compat policy for v2 vs v1 wire

### Resolution
End-to-end implementation landed (writer + reader + SGen + skip + populate). Key components:
- **Writer (`AcBinarySerializer.BinarySerializationContext.WriteStringWithDispatch`)** — single-pass UTF-8 encode + ASCII detect via `bytesWritten == charLength`; emits one of 4 markers (FixStrAscii / FixStr / StringAscii / String). Split layout for hot path: `charLength ≤ 31` encodes optimistically at `savedPos+1` (FixStr position) → 0 shift on FixStr hit; `charLength > 31` uses D-2 layout with backfill. The split avoids the post-encode left-shift that the unified layout introduced (regression seen in 12-42-32 bench).
- **Reader (`AcBinaryDeserializer.BinaryDeserializationContext.ReadAsciiBytesAsString`)** — `Encoding.Latin1.GetString` (BCL SIMD-accelerated byte→char widen). Avoids the `string.Create` callback + scalar widen overhead — measurably better on Small Deser cell (closed the +20% MemPack-relative anomaly).
- **TypeReaderTable**: `StringAscii` (167) + 32 × `FixStrAscii` (135-166) readers registered. `IsFixStrAscii` / `StringAscii` fast paths in `PopulatePropertyWithMarker`, `ReadValue`, `SkipValue`.
- **SGen (`AcBinarySourceGenerator.EmitReadString`)** — regenerated readers branch on `IsFixStr` / `IsFixStrAscii` / `case StringAscii` per property.

**Wire format version not bumped** — the new markers occupy previously-unused codepoints (135-167); old wire (without ASCII markers) is forward-compatible (readers handle both `String` and `StringAscii`). v1 stays.

**Acceptance (AOT bench 13-40-29, MemPack-relative ratios — JIT noise eliminated):**
- ✅ AcBinary Ser AND Deser GYORSABB MemPack-nél MINDEN cellán (5/5)
  - Small: Ser -8%, Deser -23%
  - Medium: Ser -17%, Deser -30%
  - Large: Ser -28%, Deser -32%
  - Repeated: Ser -4%, Deser -9%
  - Deep: Ser -24%, Deser -22%
- ✅ Wire size advantage: 2043-50419 byte (vs MemPack 3070-64986) = **-22% to -33%** across cells
- ✅ Round-trip tests: 167 pass (13 pre-existing failures are IId-tracking, unrelated to M3R7)

**JIT vs AOT note**: earlier JIT-mode benchmarks (12-50-43 → 13-27-20 series) showed elevated ratios on Small/Repeated cells (1.0-1.2 range) that disappeared under AOT publish. The JIT-mode numbers reflect tier-up artifacts (inconsistent inlining of SGen-generated reader hot paths during the 1000-iteration measurement window), not a structural M3R7 property. AOT (NativeAOT / ILC) compiles deterministically with fixed inline decisions — the steady-state numbers above reflect the actual production performance.

## ACCORE-BIN-T-E2F9: Custom UTF-8 encoder (writer-side, symmetric with custom decoder)
**Priority:** P1 · **Type:** Performance · **Related:** decoder optimization (`AcBinaryDeserializer.BinaryDeserializationContext.Read.cs::DecodeUtf8SinglePass`)
**Status:** Closed (2026-05-04)

> **Sorrendi megjegyzés:** ezt **A MARKER-DISPATCH ELŐTT** csináljuk (lásd `ACCORE-BIN-T-M3R7`). Indok: a custom encoder/decoder optimalizáció a "nehezebb, kevésbé biztos" win — a non-ASCII / mixed content workload-okat (Repeated Strings Hungarian) hozza be. A marker-dispatch utána már csak additív tisztítás a pure ASCII path dispatch-overhead-jén.

Replace `Encoding.UTF8.GetBytes` calls in `WriteStringUtf8` / `WriteStringUtf8Internal` / `WriteFixStrDirect` (collectively the writer's UTF-8 encode path, post-D-2) with a hand-rolled SIMD encoder. Symmetric to the decoder optimization (V4N2 / Read.cs::DecodeUtf8SinglePass).

### Layered structure (mirrors decoder)
- **Phase 1 — Vector256 ASCII narrow**: 16 chars (Vector256<ushort>) → 16 bytes (Vector128<byte>) via `Vector256.Narrow`. ASCII detect via `(v & 0xFF80).ExtractMostSignificantBits() == 0` (any high bit on UTF-16 char). Break on first non-ASCII char.
- **Phase 2 — DWORD ASCII batch**: 4 chars at a time, OR-mask test, 4 bytes per iter when ASCII.
- **Phase 3 — Scalar multi-byte encode**: 1-byte (ASCII) / 2-byte (Latin extended) / 3-byte (BMP) / 4-byte (surrogate pair → supplementary plane) UTF-8 encoding via direct bit-extract. No fallback dispatch — input is trusted UTF-16 (string).
- Use `System.Text.Unicode.Utf8.FromUtf16` as fallback target for scalar correctness — or skip BCL entirely with manual bit-pack.

### Why
`Encoding.UTF8.GetBytes` carries virtual-dispatch + encoder-fallback overhead even with SIMD ASCII fast path internally. Custom encoder skips this. ~15-30% Ser improvement on ASCII content, ~5-10% on non-ASCII (multi-byte path stays scalar).

### Trigger
- **NEXT** — implementation order P1 before marker-dispatch (M3R7)
- Re-evaluate if .NET 11 BCL UTF-8 GetBytes becomes faster (PR #120628 follow-up)

### Acceptance
- Writer-side benchmark: ≥15% Ser speedup on ASCII content (Small/Medium/Large/Deep), ≥5% on non-ASCII (Repeated)
- Wire format unchanged (custom encoder produces same bytes as `Encoding.UTF8`)
- Round-trip tests pass

### Resolution
Implemented as `EncodeUtf8SinglePass` in `AcBinarySerializer.BinarySerializationContext.cs` — three-phase layered encoder (Vector256 ASCII narrow + DWORD ASCII batch + scalar 1/2/3-byte BMP & 4-byte surrogate-pair). Bypasses `Encoding.UTF8.GetBytes` virtual-dispatch + encoder-fallback overhead. Trusted-input path — no validation pass on writer side (the input is a .NET `string` with valid UTF-16 surrogate pairs by construction).

Used by `WriteStringUtf8` (D-2 single-pass with VarUInt backfill) and `WriteStringWithDispatch` (M3R7 marker-dispatch path). Wire format unchanged — the encoder produces the same bytes as `Encoding.UTF8.GetBytes`.

Acceptance (per bench 12-50-43 → 13-27-20, MemPack-relative ratios on AcBinary Compact FastMode SGen):
- ✅ ASCII Ser ≥ MemPack on 4/5 cells (Small 0.94, Medium 0.80, Large 0.79, Deep 0.81)
- ⚠️ Repeated Ser ~1.04 (Hungarian, multi-byte path scalar) — see follow-up `ACCORE-BIN-T-H7K3`
- ✅ Round-trip tests pass (167 of 180; 13 pre-existing failures unrelated to encoder)

## ACCORE-BIN-T-W7N5: Default-value omission policy — doc + optional opt-out
**Priority:** P2 · **Type:** Refactor + Documentation · **Status:** Closed (2026-05-04) — Won't Fix (JIT-only artifact) · **Related:** `BINARY_ISSUES.md#accore-bin-i-d9y2` (canonical issue)

The serializer's `PropertySkip` (102) optimization saves 1 byte per default-valued property by omitting the full value from the wire — relying on the consumer-side type definition to have the same `default(T)`. This is a **latent correctness risk** documented in `ACCORE-BIN-I-D9Y2`. This entry tracks the mitigation plan; full failure-mode analysis lives in the issue.

### Decision tree (TBD when implementing)

1. **Doc-only**: position as a deliberate protobuf-style feature; consumer keeps type defaults stable across versions. Lowest cost, maximum benchmark wire-size advantage retained.
2. **Option flag**: `AcBinarySerializerOptions.OmitDefaults` boolean. Default `true` (preserves current behavior + benchmark numbers). `false` writes every property in full — opt-out for fragile-class-evolution scenarios.
3. **Both**: ship doc + flag. Default behavior unchanged; consumers who hit silent-corruption have an explicit opt-out.

### Acceptance (when implementing)

- `BINARY_FEATURES.md` adds a "Default-Value Omission" section documenting the semantic and the tradeoff (with cross-ref to `ACCORE-BIN-I-D9Y2`)
- If flag added: round-trip tests covering both `true` and `false`; benchmark comparison table showing wire-size delta on ASCII / Hungarian / DTO-heavy workloads
- Decision rationale recorded in `LLM_PROTOCOL_DECISIONS.md` (or a `### Resolution` block on the issue) once implemented

## ACCORE-BIN-T-V4N3: Symmetric `GetUtf8ByteCount` API + writer-side BCL kihagyás (cold path)
**Priority:** P3 · **Type:** Performance · **Status:** Superseded (2026-05-08, by `ACCORE-BIN-T-K7M3`) — landed Closed 2026-05-06; subsequent A/B against modern `Utf8.FromUtf16` / `Utf8.ToUtf16` showed the BCL modern API outperforms the custom transcoder on every benchmark cell, leading to full hot-path switch in K7M3 · **Related:** `EncodeUtf8SinglePass`, `WriteStringUtf8Internal`, `PropertyMetadataBase.NameUtf8`, `ACCORE-BIN-T-K7M3` (hot-path BCL switch)

Symmetric byte-count helper for `EncodeUtf8SinglePass`, paired with writer-side BCL `Encoding.UTF8.GetBytes` / `GetByteCount` removal across all cold-path call sites. `Utf8Transcoder.GetUtf8ByteCount(ReadOnlySpan<char>)` SIMD impl (Vector512 / Vector256 / Vector128 / scalar tier hierarchy, 5-popcount closed-form aggregation handling chunk-split surrogate pairs correctly).

**Implementation summary**:
- `Utf8Transcoder.GetUtf8ByteCount` SIMD impl with closed-form `bytes = 3*N - ascii - c_lt_0x800 + highSur - 3*lowSur` aggregation
- `Utf8TranscoderTests` extended (29 new tests covering ASCII / Hungarian / CJK / emoji / boundary 0-64, plus surrogate-pair-split-across-SIMD-chunks regression coverage)
- `WriteStringUtf8Internal` (`BinarySerializationContext.cs:875`) refactored from BCL two-pass to single-pass D-2 layout (worst-case `length*4` allocate + `EncodeUtf8SinglePass` + VarUInt backfill); the `4×` worst-case capacity is amortized by the buffer growth doubling strategy (`Math.Max(buffer.Length*2, position+needed)` + ArrayPool bucket-rounding to next power-of-2)
- Cold path cleanup: `AcBinarySerializer.AnalyzeStringInternCandidates` (analysis log) and `PropertyMetadataBase.NameUtf8` ctor-once init both migrated to `Utf8Transcoder`

### Resolution

Landed 2026-05-06. All `Utf8TranscoderTests` pass (55/55). Binary test suite unchanged (222 pass / 13 pre-existing GuidIId failures, untouched).

**Critical observation surfaced during the audit**: `WriteStringUtf8Internal` has only one caller (`WriteFixStrDirect`), and `WriteFixStrDirect` itself is **uncalled anywhere in the codebase** — no core call site, no SourceGenerator template hit (verified against `AcBinarySourceGenerator.cs` line 706/724/1492/1514 — generator emits `WriteStringGenerated` and `context.WriteStringUtf8` (the public 659-line method, not `WriteStringUtf8Internal`)), no test, no reflection path. The V4N3 implementation therefore landed cleanly but its hot-path benchmark impact is limited to the two cold-path init sites. Dead-code disposition tracked as `ACCORE-BIN-T-V4N5`.

**Algorithmic correctness lesson** — the initial 4-popcount formula (`3*N - c_lt_0x80 - c_lt_0x800 - 2*highSur`) was wrong on chunks where a surrogate pair straddles the SIMD chunk boundary (it implicitly assumed `lowSur == highSur` per chunk, which is true over the whole well-formed string but NOT per chunk). Fix: 5-popcount closed-form (`3*N - ascii - c_lt_0x800 + highSur - 3*lowSur`), with the scalar tail using the same per-char accounting model (`i += 1` per char regardless of role; high → 4, low → 0, BMP → 3, two-byte → 2, ASCII → 1). Caught by `GetUtf8ByteCount_MultipleEmojiBoundary_MatchesBcl` and `GetUtf8ByteCount_BoundaryAsciiToEmoji_MatchesBcl` regression tests — exactly the `prefixLen` 1, 7 boundaries that exercise chunk-split surrogate pairs.

### Superseded by `ACCORE-BIN-T-K7M3` (2026-05-08)

The V4N3 audit measured the custom transcoder against the **legacy `Encoding.UTF8.GetBytes`** API and won. **Did NOT measure against the modern `System.Text.Unicode.Utf8.FromUtf16` / `Utf8.ToUtf16` static API** (.NET 7+, used by MemoryPack source-gen). Once `D9X3` stabilized the bench, a direct A/B revealed the BCL modern API outperforms the custom transcoder on **every** cell (Ser deficit -14 to -22pp, Deser flips from behind to ahead). All 8 hot-path call sites switched to BCL in `K7M3`. The `Utf8Transcoder.cs` file is fully commented out — preserved as historical reference.

The V4N3 algorithmic correctness work (5-popcount surrogate-pair-split-across-chunks closed-form) remains a **valid algorithmic contribution**, but no longer load-bearing on the hot path.

## ACCORE-BIN-T-J5L9: Remove dead `WriteFixStrDirect` / `WriteStringUtf8Internal` (audit-surfaced uncalled methods)
**Priority:** P3 · **Type:** Refactor / hygiene · **Status:** Closed (2026-05-06) · **Related:** `BinarySerializationContext.cs`

V4N3 audit surfaced two methods with no callers in the entire workspace:

- `WriteFixStrDirect(string)` — public method, no call site (no core, no SourceGenerator template, no test, no reflection / Expression-compile)
- `WriteStringUtf8Internal(string)` — private method called only from `WriteFixStrDirect`'s non-ASCII fallback branch

The pair forms a closed dead loop (`WriteFixStrDirect` → `WriteStringUtf8Internal`), but no entry point reaches `WriteFixStrDirect`. The public-API `WriteStringUtf8` (line 659) is the live equivalent and is called from the SourceGenerator template (polymorphism path: assembly-qualified type-name write). The hot-path string-write goes through `WriteStringWithDispatch` (line 734) which uses the M3R7 marker-dispatch — NOT through this dead pair.

### Disposition options (decide pre-NuGet release)

1. **Delete both methods** — pure dead-code cleanup; reduces public surface, removes maintenance burden, simplifies onboarding. Functionality is fully covered by `WriteStringWithDispatch` (M3R7 marker-dispatch — emits `FixStr` / `FixStrAscii` directly with proper ASCII detection via `bytesWritten == charLength` after `EncodeUtf8SinglePass`).
2. **Activate `WriteFixStrDirect` for property-name writes** — SGen could emit `WriteFixStrDirect(propName)` instead of `WriteStringWithDispatch(propName)` for known-short, often-ASCII property names — saving the marker-dispatch overhead. Requires SGen template change + benchmark validation that the saving is real (likely marginal — property names are typically <31 char ASCII, so M3R7 already takes the FixStrAscii fast path with one byte-write to `_buffer`). The pre-encoded `NameUtf8` byte[] on `PropertyMetadataBase` already provides a faster path (`WriteFixStrBytes` at line 853) which the SGen / runtime writer could use directly.
3. **Defer** — leave as-is, document as dead code, revisit when the codebase has another reason to touch this area.

### Why P3

- No correctness or perf impact in either direction (dead code is dead — no consumer affected)
- Cleanup vs activation is a low-stakes choice; benchmark would decide if option 2 has real saving
- Surfaced during V4N3 work, not blocking the NuGet release

### Acceptance

- Decision recorded (delete / activate / defer) with rationale
- If "delete": grep across workspace confirms zero callers post-removal; binary test suite unchanged (still 235 pass / 13 pre-existing failures)
- If "activate": SGen template change + benchmark validation showing ≥ 2% Ser improvement on a representative cell (otherwise revert to "delete")
- Documentation in `BINARY_IMPLEMENTATION.md` updated (or remove the old reference if both methods deleted)

### Trigger

- Pre-NuGet release housekeeping pass
- Or: any future refactor that touches `BinarySerializationContext` string-write methods (then decide rather than leave the dead pair behind)

### Resolution

Disposition: **Delete (Option 1)**. Landed 2026-05-06 together with the H2Q6 marker reorg commit. Five dead methods removed in a single cleanup pass:

- `WriteFixStrDirect(string)` — uncalled public method
- `WriteStringUtf8Internal(string)` — uncalled private method (only called from `WriteFixStrDirect`)
- `WriteFixStr(string)` — uncalled public method (audit surfaced; was originally listed as live)
- `WriteFixStrBytes(ReadOnlySpan<byte>)` — uncalled public method (audit surfaced)
- `WritePreencodedPropertyName(ReadOnlySpan<byte>)` — uncalled public method (audit surfaced)

All five had zero call sites across core, SourceGenerator template, tests, and reflection. The hot-path string write continues through `WriteStringWithDispatch` (M3R7 + H2Q6 marker dispatch) and `WriteStringInternFirstWithDispatch` (interning tier dispatch). Public surface reduced; binary test suite unchanged (222 pass / 13 pre-existing GuidIId failures).

## ACCORE-BIN-T-O7G2: Overflow guard on `charLength * 4` writer arithmetic + corrupted-wire `ReadStringBig`
**Priority:** P3 · **Type:** Defensive / safety · **Status:** Closed (2026-05-06) · **Related:** `WriteStringWithDispatch`, `WriteStringInternFirstWithDispatch`, `ReadStringBig`, `BinaryTypeCode.MaxStringCharLength`

Defensive guards covering two latent failure modes in the H2Q6 string serialization paths:

**Writer overflow (silent zero corruption)** — `charLength * 4` overflows `int` when `charLength > 0x1FFFFFFF` (~537M). At exactly `0x40000000` chars the multiplication wraps to **0**, causing:
- `EnsureCapacity(reserveHeader + 0)` to silently succeed (no buffer growth)
- `EncodeUtf8SinglePass(value, emptySpan)` to write 0 bytes, returning `bytesWritten = 0`
- The H2Q6 tier choice picks Small (`bytesWritten ≤ 255`), writing `[StringSmall][0][0]` to the wire
- **The string content is lost silently — no exception, wire claims an empty string**

Other overflow values (e.g. `charLength = 600M` → `maxBytes` becomes negative) eventually surface as `ArgumentOutOfRangeException` from `Span.AsSpan(start, length)`, but the message ("length cannot be negative") is misleading and arrives after the buffer has already been partially mutated.

**Reader corrupted wire (negative cast from oversized uint)** — in `ReadStringBig`, the wire-side `charLen:32` and `utf8Len:32` are read as `uint`, then cast to `int`. Corrupted or maliciously-crafted payloads with values > `Int32.MaxValue` produce negative ints, leading to `string.Create(negative, ...)` exceptions or position-state desync — at best a misleading message, at worst a partial decode with wire-position shifted incorrectly.

### Resolution

Landed 2026-05-06 (this commit window).

**Writer side** — `WriteStringWithDispatch` and `WriteStringInternFirstWithDispatch` each gain one method-entry guard:

```csharp
var charLength = value.Length;
if ((uint)charLength > BinaryTypeCode.MaxStringCharLength) ThrowStringTooLong(charLength);
```

A single unsigned compare catches the overflow band; predict-friendly (always false on realistic input). The throw helper is `[MethodImpl(MethodImplOptions.NoInlining)]` so the JIT/AOT keeps the throw site out of the inlined hot path. The same `charLength` value is reused across the FastWire and Compact branches — no duplicate guard.

**Reader side** — `ReadStringBig` gains a single bitwise-OR + sign-test:

```csharp
var packed = context.ReadUInt64Unsafe();
var charLength = (int)(uint)packed;
var byteLength = (int)(uint)(packed >> 32);
if ((charLength | byteLength) < 0) ThrowCorruptedBigWire(charLength, byteLength);
```

The OR + sign-test catches negative casts (any wire-side uint > `Int32.MaxValue` produces a negative int after cast; OR of two positives is positive, sign-test cheap). One instruction effective; predict-friendly.

**New constant**: `BinaryTypeCode.MaxStringCharLength = 0x1FFFFFFF` (536_870_911 — largest charLength where `charLength * 4` fits in int).

**Hot-path cost**: ~0% on realistic input — single unsigned compare on the writer, single OR + sign-test on the reader Big tier (Small/Medium readers untouched since their wire values are bounded by `byte` / `ushort` types and cannot overflow). Throw helpers `NoInlining` keep the inlined caller body compact. Tests 222 pass / 13 pre-existing failures unchanged.

### Why P3

- No correctness impact for realistic inputs (the overflow band is far outside any real DTO scenario)
- Defensive value: prevents silent data loss in the `charLength = 1.07G` zero-overflow edge case + provides clear error messages on out-of-range inputs
- Security value: corrupted/malicious wire payloads on the reader Big tier path are now caught early instead of producing inconsistent position state
- NuGet release professional-quality signal — explicit, defensive guards over silent-corruption paths

## ACCORE-BIN-T-C5R8: Charset-parameterized benchmark workload (ASCII / Hungarian / CJK / Cyrillic / Mixed)
**Priority:** P2 · **Type:** Tooling / release-narrative · **Status:** Closed (2026-05-07) · **Related:** `BenchmarkTestDataProvider`, `AyCode.Core.Serializers.Console.Program.cs` (Settings → Charset submenu), `ACCORE-BIN-T-V4N2` (charset-specific optimization measurement target), `ACCORE-BIN-T-D9X3` (bench stabilization preceding this work)

The current `BenchmarkTestDataProvider` hard-codes Hungarian (Latin extended 2-byte) content into the test DTOs. This produces a single workload-shape: **Hungarian mixed text with short 1-2 char 2-byte runs**. While Hungarian is a fine general-purpose i18n stress, it is **only one production-content profile** — and the optimization decisions ride on it implicitly (e.g. V4N2 Phase 2.5's 3-byte run do-while was deferred-on-2-byte-side because the Hungarian bench measured regression there, but its CJK-side value cannot be measured on the current data).

A **charset-parameterized** benchmark workload — selectable from the interactive menu — would:

- **Measure optimization value across realistic content profiles** — what wins on CJK content may not win on Hungarian, and vice versa. Without explicit per-charset measurement, optimization decisions become Hungarian-biased.
- **Surface release-narrative numbers credibly** — instead of "Compact beats MemPack on i18n payload" (single workload), claim "Compact vs MemPack: ASCII X%, Hungarian Y%, CJK Z%, Cyrillic W%, Mixed V%" — concrete numbers per content profile, NuGet-grade.
- **Enable workload-specific optimization audits** — V4N2 Phase 3 SIMD multi-byte transcoder targets CJK 3-byte content; without a CJK workload measurement, Phase 3 acceptance criteria cannot be validated.

### Implementation outline

#### 1. `BenchmarkTestDataProvider` refactor

Hard-coded Hungarian strings (`KözösCímke`, `sötét`, `magyar`, `hetenkénti`, etc.) → **ASCII baseline values** (English equivalents: `SharedTag`, `dark`, `hungarian`, `weekly`).

New static `LongStringSuffix` field — charset-aware suffix appended to a subset of property values:

```csharp
public static class CharsetSuffixes
{
    public const string AsciiOnly = "";  // baseline — pure-English ASCII content
    public const string Hungarian  = " árvíztűrő tükörfúrógép";
    public const string CjkBmp     = " 你好世界 こんにちは 안녕하세요";
    public const string Cyrillic   = " Привет мир дорогой друг";
    public const string Mixed      = " árvíz 你好 Привет 😀";
}

public static string LongStringSuffix { get; set; } = CharsetSuffixes.Hungarian;  // default
```

Property values use the suffix dynamically:
```csharp
var description = "Product description" + LongStringSuffix;
```

The 5 charsets cover the realistic UTF-8 workload spectrum:
1. **Pure ASCII** — baseline; Phase 1 SIMD prefix widen + DWORD batch dominate; no multi-byte path engagement
2. **Hungarian** (Latin extended) — short 1-2 char 2-byte runs in mixed text; current default workload
3. **CJK BMP** — long homogeneous 3-byte runs; primary V4N2 Phase 2.5/3 win region
4. **Cyrillic** (Russian / etc.) — long 2-byte runs (different shape than Hungarian mixed); V4N2 Phase 2.5 may yet pay off here
5. **Mixed** (Hungarian + CJK + emoji) — full multi-tier coverage in one payload; surrogate-pair handling stress

#### 2. `Program.cs` interactive submenu

Before starting a benchmark run, prompt the user for charset choice:

```
Choose benchmark charset:
  1 — Pure ASCII (baseline)
  2 — Hungarian (Latin extended) [DEFAULT]
  3 — CJK BMP (Chinese / Japanese / Korean)
  4 — Cyrillic (Russian / etc.)
  5 — Mixed (Hungarian + CJK + emoji)
```

The choice → `BenchmarkTestDataProvider.LongStringSuffix = ...` before constructing test data.

#### 3. Benchmark output header

The markdown output header should reflect the selected charset:
```
# AcBinary Benchmark Release 2026-05-07 16:00:00
Charset: CJK BMP | Iterations: 1000 | Warmup: 10000 | ...
```

This makes per-charset bench files self-documenting — file names + content both encode the workload profile.

#### 4. Round-trip tests unaffected

`Utf8TranscoderTests` and other content-class unit tests (with their fixed Hungarian / CJK / emoji boundary inputs) are **untouched** — they remain fixed-content for regression coverage. Only the benchmark workload is charset-parameterized.

### Why P2

- **Release-narrative**: NuGet release credibility depends on measurable performance claims across realistic content profiles, not a single Hungarian-mixed workload
- **Optimization decision quality**: V4N2 Phase 2.5 / Phase 3 / future SIMD multi-byte work cannot be objectively validated without a CJK workload — current decisions have implicit Hungarian-bias
- **Consumer reproducibility**: external consumers can reproduce benchmark numbers on their own content profile (or contribute a new charset profile)

### Acceptance

- `BenchmarkTestDataProvider` refactored: ASCII baseline + `LongStringSuffix` static field with 5 predefined charset constants
- Interactive menu in `Program.cs` lets the user choose charset 1-5 before benchmark run; the chosen charset is recorded in the markdown output header
- Round-trip correctness verification still runs once-per-cell before warmup (existing `Verified: round-trip ...` line) — works on the active charset
- All 5 charsets produce valid round-trip on all benchmark cells (Small / Medium / Large / Repeated / Deep)
- Existing benchmark numbers (Hungarian-default) reproducible — choosing charset 2 from the menu yields the current 15:29:21-style results
- New CJK charset (option 3) produces measurable numbers (one bench run per charset documented in `Test_Benchmark_Results/`)

### Trigger

- Pre-NuGet release: per-charset numbers needed for the public performance-claim table
- Or: when V4N2 Phase 3 SIMD multi-byte transcoder work needs CJK-workload validation

### Resolution

Landed 2026-05-07 (after `ACCORE-BIN-T-D9X3` bench stabilization made sub-3% deltas measurable, which raised the value of charset-specific measurement). Implementation refined the original 5-charset proposal into a 6-charset list per user request (Latin1FixAscii + Latin1 short/long split for finer-grained Latin1 coverage):

**1. `BenchmarkTestDataProvider` refactor** ✅

- New `CharsetSuffixes` static class with **6** const suffixes (one more than originally proposed):
  - `Latin1FixAscii = ""` — empty suffix; baseline values stay short → FixStr fast-path stress (renamed from `AsciiOnly` per user request)
  - `Latin1Short = " árvíztűrő tükörfúrógép"` (~24 char) — Hungarian short Latin1 mixed
  - `Latin1Long = " árvíztűrő tükörfúrógép a magyar betűzés tesztje"` (~47 char) — **NEW**, exceeds the 32-char FixStr boundary on the suffix alone (user request)
  - `CjkBmp`, `Cyrillic`, `Mixed` — as originally specified
- `LongStringSuffix` default = `CharsetSuffixes.Latin1Long` (backward-compatible in spirit with the prior fixed Latin1 default)
- All hard-coded Hungarian baseline values replaced with ASCII English equivalents:
  - `KözösCímke` / `IsmétlődőCímke` / `MélyCímke` → `SharedTag` / `RepeatedTag` / `DeepTag`
  - `közösfelhasználó` → `shareduser` (and variants); `közös` → `shared`; `MélyKategória` → `DeepCategory`
  - `sötét` / `világos` → `dark` / `light`; `magyar` / `német` / `francia` → `hungarian` / `german` / `french`
  - `hetenkénti` / `naponkénti` / `havonkénti` → `weekly` / `daily` / `monthly`
  - Repeated cell long Hungarian baselines (`TermékNév_IsmétlődőTesztAdat_árvíztűrőtükörfúrógép`, `RaklapKód_IsmétlődőTesztAdat_árvíztűrő`) shortened to ASCII `ProductName` / `PalletCode` so the `EnsureAllStringsBypassFixStr` suffix-append actually applies (the prior >31-char baselines bypassed the suffix, leaving Repeated cell content fixed-Hungarian regardless of charset selection)
- The only Latin1/non-ASCII characters remaining in the file are inside the `CharsetSuffixes` const definitions themselves (intentional — those define the per-charset content profiles)

**2. `Program.cs` interactive submenu** ✅

- New `[3] Charset` entry in the existing `Settings` submenu (next to `[1] Iteration` and `[2] WireMode`) — chose nested submenu over a top-level prompt to keep the main menu uncluttered
- `ShowCharsetSettingsMenu` lists the 6 charset constants with brief descriptions; selection sets `BenchmarkTestDataProvider.LongStringSuffix` and returns
- `GetCurrentCharsetName()` helper resolves the active suffix back to its constant name (returns `"Custom"` when programmatically set to a non-const value)

**3. Benchmark output header** ✅

- `Charset:` field added to **3 output locations**:
  - Console run header (interactive run line — `Layer: ... | Charset: CjkBmp | Iterations: ...`)
  - `.LLM` markdown header (file-self-documenting)
  - `.log` boxed banner (║  Charset: CjkBmp  ║)

**4. Round-trip tests unaffected** ✅ — `Utf8TranscoderTests` and other content-class unit tests use their own fixed boundary inputs; not touched by this change. Round-trip verification in the bench harness continues to run once-per-cell pre-warmup (`VerifyRoundTrip`) on the active charset.

### Acceptance status

- ✅ `BenchmarkTestDataProvider` refactored with ASCII baselines + `LongStringSuffix` field + 6 charset constants
- ✅ Interactive submenu lets the user choose charset 1-6; recorded in markdown output header (3 locations)
- ✅ Round-trip verification runs on the active charset (existing per-cell verify, charset-agnostic by design)
- ⚠️ "All 6 charsets produce valid round-trip on all benchmark cells" — design correctness implies this; not yet exercised on every (cell × charset) combination explicitly. Recommend running each charset once before declaring full validation.
- ❌ "Existing benchmark numbers (Hungarian-default) reproducible — choosing charset 2 yields the current 15:29:21-style results" — **NOT met**: the ASCII baseline refactor changes the numbers regardless of charset choice (shorter baselines + suffix-driven content vs. prior fixed Hungarian baselines). New `Latin1Short` ≠ prior fixed Hungarian default. This is intentional: the user explicitly chose a clean ASCII-baseline + charset-suffix design over preserving historical numerical comparability.
- ❌ "Choosing CJK produces measurable numbers documented in `Test_Benchmark_Results/`" — **NOT done in this commit window**; user has the menu and will run per-charset benches in a follow-up sprint.

### Note on numerical incompatibility with prior runs

Existing bench files generated before this commit (e.g. `Console.FullBenchmark_Release_2026-05-07_17-42-22.LLM` and earlier) used the prior fixed Latin1 baseline values + 32-char Hungarian suffix. The new default (`Latin1Long`) uses ASCII baselines + 47-char Latin1Long suffix; the Repeated cell sees a more dramatic shift (its 52-char fixed Hungarian baseline → 11-char ASCII `ProductName` + 47-char suffix). **Numerical comparison across the boundary is not meaningful**; the `Charset:` header field documents the source charset for each new bench file.

### Future extensions

- **Sentinel "real-world" charsets** — synthetic mixes representing typical production payloads (e.g. `EnglishWithEmoji` for chat-app DTOs, `ArabicHebrew` for RTL-script regions). Add as new `CharsetSuffixes` constants when consumer demand surfaces.
- **Charset auto-rotate mode** — single benchmark run cycles through all 5 charsets, producing a 5-section markdown output. Useful for full release-narrative table generation in one pass.
- **BDN integration** (per `ACCORE-BIN-T-B1D5`): charset becomes a `[Params]` axis in BenchmarkDotNet, producing a 5×5×N matrix (cells × charsets × engines) in the BDN output.

## ACCORE-BIN-T-D9X3: Console benchmark stabilization (per-serializer warmup + GC isolate + pilot discard + min/max range + CPU pin + mode-aware JIT sleep)
**Priority:** P1 · **Type:** Tooling / measurement · **Status:** Closed (2026-05-07) · **Related:** `AyCode.Core.Serializers.Console.Program.cs`, `ACCORE-BIN-T-V4N4`, `ACCORE-BIN-T-V4N2`, `ACCORE-BIN-T-S6F2`, `ACCORE-BIN-T-B1D5` (BDN release-grade variant)

The custom `Console` benchmark harness showed strong run-to-run variance — user-reported `±20pp / -10pp` summa-spread between runs on identical code. 1-3% perf-claims became unmeasurable on this noise-floor; the V4N4 method-split and V4N2 Phase 2.5 attempts both fell into this band, leaving the question "does the regressed bench number reflect a code regression or measurement noise?" undecidable (see `V4N4` Reverted section).

**Diagnosis** (sprint takeaway prior to this entry):

1. **Warmup cache pollution** — `RunBenchmarksForTestData` ran one warmup-all loop (every serializer × WarmupIterations) followed by one bench-all loop. By the time a given serializer was measured, its hot code and data lines had been evicted by the intervening serializers' warmup passes. MemPack and AcBinary hot paths share neither code nor data working sets — they actively evict each other.
2. **GC pause leakage between samples** — the Stopwatch-recorded sample loop had no explicit `GC.Collect`. A minor GC triggered inside sample N could promote into a Gen-2 pause inside sample N+1's timed window (1-5 ms spike).
3. **Pilot sample contamination** — the first sample after warmup absorbed residual JIT bookkeeping and cold-cache misses; on a 10-sample median this contributed 1-2 outliers that visibly stretched the min/max.
4. **CPU migration / preemption** — the Windows scheduler migrated the bench thread between cores between samples (L1/L2 cache evict on each migration); background work (Defender index, OS service threads) injected random preemption spikes.
5. **JIT sleep not mode-aware** — `Thread.Sleep(JitSleep = 3000)` waited 3 seconds before each cell for tiered-JIT drain. On AOT publish (`PublishAot=true`) there IS NO dynamic compilation — the 3 seconds were pure idle. Worse, the drain happened only globally (once before all cells), not per-serializer, so a tier-promotion mid-bench could still bleed in.
6. **Range invisible** — the `.LLM` markdown output showed only the median; the user could not tell whether a 5%-median-delta was inside or outside the inter-sample range for that row.

### Resolution

Landed 2026-05-07 (16:00 — 17:00). Six stabilization steps in one commit window:

**1. Per-serializer warmup separation** (`RunBenchmarksForTestData`) — the warmup-loop and bench-loop merged into one per-serializer cycle: each serializer's warmup runs IMMEDIATELY before its own bench. The serializer's hot code/data is freshest in cache when the first sample times.

**2. `GC.Collect` before every sample** (`RunTimed`) — `GC.Collect() + WaitForPendingFinalizers() + GC.Collect()` triple-tap before each sample, OUTSIDE the Stopwatch window. Every sample starts from the same heap state; an ad-hoc Gen-2 pause from sample N can no longer bleed into sample N+1.

**3. Pilot sample discard** (`RunTimed`) — the loop runs `samples + 1` times; the first (index 0) is discarded. The first sample post-warmup absorbs residual JIT/GC bookkeeping and cold cache; the recorded `samples` count remains 10 (median is the same data the user saw before, just sourced from "typical" sample-set, not from the post-warmup-first noisy point).

**4. Min/max range in markdown output** (`SaveLlmResults`, new `FormatMicrosWithRange` helper, new `BenchmarkResult` fields: `SerializeTimeMinMs/MaxMs`, `DeserializeTimeMinMs/MaxMs`, `RoundTripTimeMinMs/MaxMs`) — the `.LLM` output's Ser and Deser columns now render as `26.86 (24.50..29.10)`: median (min..max) µs/op. The reader sees at a glance whether a delta is above the row's noise floor.

**5. CPU affinity + process priority** (`RunBenchmark`) — `ProcessorAffinity = 0x1` (CPU 0 pin) + `PriorityClass = High` for the benchmark phase, `try/finally` restores the original values. Eliminates inter-sample thread migration (L1/L2 cache evicts) and reduces background-task preemption. Platform-guarded: Windows / Linux only (`CA1416` — `ProcessorAffinity` throws on macOS); locked-down hosts (group policy, container without `CAP_SYS_NICE`, etc.) catch + warning + bench continues with default scheduling.

**6. Mode-aware `JitSleep`** (property) — `RuntimeFeature.IsDynamicCodeCompiled ? 250 : 0`. JIT mode 250 ms (the .NET 9 tiered-JIT compile queue typically drains in <100 ms for the bench's hot path); AOT publish 0 ms. The 3000 ms blind wait is gone. The drain now happens per-serializer (Step 1) instead of once globally.

### Bench result (3 consecutive runs, 2026-05-07 17:00:32 / 17:01:03 / 17:01:32, FastestByte mode, FastMode preset)

| Cell | AcBinary Ser median (3 runs) | Inter-run spread | Intra-cell range |
|---|---|---|---|
| Small | 7.09 / 6.83 / 6.55 | 7.6% | ~8% (noise floor: 1000×6ns measured) |
| Medium | 18.74 / 18.90 / 19.22 | 2.6% | ~10% |
| Large | 140.20 / 141.67 / 141.02 | 1.0% | ~3% |
| Repeated | 26.52 / 26.25 / 26.28 | 0.3% | ~6% |
| Deep Nested | 23.44 / 23.17 / 22.70 | 3.2% | ~7% |

The previous `±20pp / -10pp` summa-spread shrank to **1-3pp** on the medium/large cells. The Small cell remains noisy (~8% relative) but this is a physical floor: 1000 iter × 6 ns/op = 6 µs total batch — below this, Stopwatch resolution and OS spikes dominate relatively.

The `(min..max)` range is consistently 3-10% relative — a **measurable** signal floor: 1-3% perf-deltas no longer disappear into noise.

### Lessons

- **Bench stabilization is a precondition for perf optimization, not a consequence.** Optimization decisions (e.g. V4N4 method-split, V4N2 Phase 2.5) can only be derived from bench numbers if the noise floor < expected signal. Without that, the bench numbers mean nothing.
- Cache pollution (warmup-all → bench-all flow) was the **single largest** noise source: per-serializer warmup separation alone removed ~10pp of variance.
- Platform stabilization (CPU pin + high priority) combined with heap stabilization (GC.Collect + pilot discard) further tightened the range.
- AOT and JIT have different stabilization needs: the 3000 ms blind sleep was idle time on AOT; mode-aware sleep pays the cost only when needed.

### Re-evaluation list (entries currently Reverted or unmeasurable)

The stabilization opens a follow-up sprint: the `Reverted (2026-05-07)` entries are re-evaluable now that the noise floor < the expected 1-3% signal:
- **`ACCORE-BIN-T-V4N4`** — method-split (writer + reader hot path) is re-testable
- **`ACCORE-BIN-T-V4N2` (Phase 2.5)** — UTF-8 do-while runs (2-byte / 3-byte) per charset
- **`ACCORE-BIN-T-S6F2`** — Small fast path (was integrated into V4N4)

Per-entry re-evaluation is the next sprint's task, NOT part of this Closed entry.

### Why P1

- Blocked all sub-3% perf optimization work (every recent attempt fell into the noise band)
- One-line user complaint ("+20 és -10 között ingadozott a summa") summarized weeks of unproductive bench-driven investigation
- One-time fixed cost; every future bench run benefits

### Follow-up: adaptive iteration + CV reporting + per-cell A/B mode (2026-05-07, second commit window)

After the initial 6-step landing, three additional refinements were added in a second commit window the same day. The trigger was a Copilot-suggested noise-reduction list against the now-stable bench output:

**1. Per-cell adaptive iteration** — fixed `TestIterations = 1000` produced sample windows from 6 ms (Small cell @ 6 ns/op) to 140 ms (Large cell @ 140 µs/op). The Small cell at 6 ms remained the dominant residual noise source (7.6% inter-run spread vs ≤3.2% on the other cells) because OS-level spikes (preempt + IRQ + scheduler tick) are absolute-time events; on a 6 ms sample window their relative contribution is huge.

Implementation:
- New constant `TargetSampleMs = 250` (per-sample wall-clock target)
- New helper `CalibrateIterations(Action, int targetMs)` — runs a 100-iter probe post-warmup, computes `iterPerMs`, and rounds up to the nearest 1000. Floor 1000, ceiling 200_000.
- `RunBenchmarksForTestData` calibrates Ser and Des INDEPENDENTLY per serializer (different per-op cost). RT-only rows (NamedPipe) get a single RT calibration.
- New `BenchmarkResult` fields: `SerializeIterations`, `DeserializeIterations`, `RoundTripIterations` (per-row).
- New helpers: `ToPerOpMicros(double, int)` (replaces 1-arg variant), `SerPerOp(r)` / `DesPerOp(r)` / `RtPerOp(r)` for per-op µs from the result.
- All `Average(r => r.*TimeMs)` and `OrderBy(r => r.RoundTripTimeMs)` call-sites refactored to use per-op µs (iter-independent) — mixing batch-time across rows with different iter counts would be meaningless. ~20 call-sites total.
- RT for in-mem rows synthesized so `RtPerOp(r) == SerPerOp(r) + DesPerOp(r)` regardless of `serIter != desIter`: `RoundTripIterations = max(serIter, desIter)`, `RoundTripTimeMs = rtPerOpMicros / 1000 * RoundTripIterations`.

**Expected impact**: Small cell sample window 6 ms → ~240 ms; inter-run spread 7.6% → ~1-2% (matching the other cells). Total suite duration ~50 s → ~110-130 s.

**2. CV (coefficient of variation) reporting + unstable-row marker** — the median + (min..max) range surfaces shape but not a single-number stability metric. The CV (= stddev/mean) is the standard statistical measure; rows with CV > threshold are flagged with a ⚠️ suffix in the markdown output so a small inter-engine delta on a high-CV row is immediately obvious as noise-suspect.

Implementation:
- New constant `UnstableCVThreshold = 0.03` (3% — reasonable for stabilized in-memory benchmarks)
- `RunTimed` return tuple extended: `(median, min, max, stddev)`. Stddev computed over the (samples − pilot) population using `Math.Sqrt(Math.Max(0, E[X²] - E[X]²))`.
- New `BenchmarkResult` fields: `SerializeTimeStdDevMs`, `DeserializeTimeStdDevMs`, `RoundTripTimeStdDevMs`.
- `FormatMicrosWithRange` extended: `26.86 (24.50..29.10)` stays the default; `26.86 (24.50..29.10) ⚠️5.2%` appears when CV exceeds the threshold.

**3. Per-cell A/B mini-suite filter** — optimization-iteration loops often need only one specific cell (e.g. "tuning the Repeated cell for Hungarian charset"). The full 5-cell × 2-engine × 4-measurement suite is overkill for that.

Implementation:
- `FilterByLayer` extended: new `small` / `medium` / `large` / `repeated` / `deep` modes — case-insensitive prefix match on `TestDataSet.Name`
- `TryParseCliArgs` recognizes the new tokens: `dotnet run -- repeated` runs only the Repeated Strings cell
- `fastestbyte` mode (existing — only AcBinary FastMode + MemoryPack head-to-head) is orthogonal and stacks: `dotnet run -- repeated fastestbyte`

### Markdown output schema change

The `## Results` table gains an `Iter Ser/Des` column at the right edge — visible verification that each row's batch landed near the `TargetSampleMs` window. RT-only rows show a single `Iter` value (the RT calibration count); in-mem rows show `serIter / desIter`.

Header line updated:
- Before: `Iterations: 1000 | Warmup: 10000 | Samples: 10 (median) | ...`
- After: `Iterations: per-cell adaptive (target ~250 ms/sample) | Warmup: 10000 | Samples: 10 (median) + 1 pilot discarded | ... | UnstableCV threshold: 3%`

## ACCORE-BIN-T-K7M3: Hot-path UTF-8 transcoder switch — `Utf8Transcoder` → BCL `Utf8.FromUtf16` / `Utf8.ToUtf16`
**Priority:** P1 · **Type:** Performance · **Status:** Closed (2026-05-08) · **Related:** `ACCORE-BIN-T-V4N3` (custom transcoder origin), `ACCORE-BIN-T-V4N2` (Phase 3 SIMD multi-byte), `ACCORE-BIN-T-V4N4` (Reverted method-split), `ACCORE-BIN-T-D9X3` (bench stabilization that made the comparison measurable)

The custom `Utf8Transcoder` (V4N3) was originally implemented to bypass `System.Text.Encoding.UTF8.GetBytes` virtual-dispatch + EncoderFallback overhead. The V4N3 audit measured wins vs. the **legacy `Encoding.UTF8`** API. **What it did NOT measure**: the modern `System.Text.Unicode.Utf8.FromUtf16` / `Utf8.ToUtf16` API (.NET 7+, tier-1 optimized, used by MemoryPack `WriteUtf8` / `ReadUtf8` paths internally). Once the bench stabilized (D9X3), a direct A/B comparison surfaced that the BCL modern API consistently outperforms the custom transcoder on the binary serializer's hot path.

### Bench A/B (Latin1Long charset, FastMode SGen Compact)

| Cell | Ser delta vs MemPack — custom (`EncodeUtf8SinglePass`) | Ser delta vs MemPack — BCL (`Utf8.FromUtf16`) | Improvement |
|------|--------------------------------------------------------|------------------------------------------------|-------------|
| Small | +28.5% | +7.3% | **-21pp** |
| Medium | +23.8% | +3.1% | -21pp |
| Large | +19.6% | +5.1% | -14pp |
| Repeated | +28.8% | +10.9% | -18pp |
| Deep | +23.1% | +0.6% | -22pp |

| Cell | Deser delta vs MemPack — custom (`DecodeUtf8SinglePass`) | Deser delta vs MemPack — BCL (`Utf8.ToUtf16`) | Improvement |
|------|---------------------------------------------------------|------------------------------------------------|-------------|
| Small | +17.6% | -1.2% (paritás) | -19pp |
| Medium | +12.8% | -4.7% (AcBinary nyer) | -17pp |
| Large | +4.9% | -10.3% (AcBinary nyer) | -15pp |
| Repeated | +16.9% | -1.6% (paritás) | -18pp |
| Deep | +7.0% | -9.0% (AcBinary nyer) | -16pp |

The Deser side flipped from "consistently behind" to "wins on 3 of 5 cells, paritás on 2". The Ser side closed the deficit from +20-29% to 0-11%. **Both sides** measurable improvement on **every** cell.

### Why the custom transcoder lost

The V4N3 implementation included a 4-tier SIMD ASCII prefix path (Vector512BW / Vector256 / Vector128 / scalar) plus a DWORD ASCII batch + scalar 4-branch multi-byte fallback. **All correct, all SIMD-tuned**. But:

1. **`Utf8.FromUtf16` is also SIMD-tuned in .NET 9** — the .NET team rewrote it on top of `System.Text.Unicode.Utf8` primitives that share infrastructure with `Ascii.IsValid` / `Latin1.GetString`. AOT-publish-friendly, branch-friendly, no virtual dispatch (the `Utf8` API is static, not via an `Encoding` instance with virtual-method-table).
2. **The custom transcoder's ASCII prefix path bails out on first non-ASCII byte** — on multi-byte content (Latin extended / Cyrillic / CJK) the SIMD path runs only for the leading ASCII span, then the entire remainder falls into per-char scalar 4-branch dispatch. The BCL `Utf8.FromUtf16` SIMD-batches multi-byte content too (different algorithm — the BCL doesn't bail on first non-ASCII).
3. **AOT inline budget**: the custom transcoder's body grew with the V4N3 / V4N4 / V4N5 additions; in NativeAOT publish the call sites in `WriteStringWithDispatch` / `ReadString*` did NOT inline (V4N4 disasm audit confirmed). The BCL `Utf8.FromUtf16` is a single static method with a tighter call-site footprint.

### Resolution

Landed 2026-05-08. The 8 production hot-path call sites of `Utf8Transcoder.*` switched to BCL:

| File / line | Before | After |
|---|---|---|
| `AcBinarySerializer.cs:120` | `Utf8Transcoder.GetUtf8ByteCount` | `Encoding.UTF8.GetByteCount` |
| `AcBinarySerializer.BinarySerializationContext.cs:694` | `Utf8Transcoder.EncodeUtf8SinglePass` | `Utf8.FromUtf16(...)` |
| `AcBinarySerializer.BinarySerializationContext.cs:784` | `Utf8Transcoder.EncodeUtf8SinglePass` | `Utf8.FromUtf16(...)` |
| `AcBinarySerializer.BinarySerializationContext.cs:901` | `Utf8Transcoder.EncodeUtf8SinglePass` | `Utf8.FromUtf16(...)` |
| `AcBinaryDeserializer.BinaryDeserializationContext.Read.cs:523` | `Utf8Transcoder.CountUtf8Chars` | `Encoding.UTF8.GetCharCount` |
| `AcBinaryDeserializer.BinaryDeserializationContext.Read.cs:527` | `Utf8Transcoder.DecodeUtf8SinglePass` | `Utf8.ToUtf16(...)` |
| `AcBinaryDeserializer.BinaryDeserializationContext.Read.cs:565` | `Utf8Transcoder.DecodeUtf8SinglePass` | `Utf8.ToUtf16(...)` |
| `PropertyMetadataBase.cs:104-109` (ctor-once) | `Utf8Transcoder.GetUtf8ByteCount` + `EncodeUtf8SinglePass` (two-pass) | `Encoding.UTF8.GetBytes(string)` (single-pass with exact-size byte[] return) |

The count-only call sites (`GetByteCount` / `GetCharCount`) stay on the **legacy** `Encoding.UTF8` API — `System.Text.Unicode.Utf8` has no count-only equivalent (only `FromUtf16` / `ToUtf16` which encode + count combined). For pure count, the legacy API is the optimal tool (single SIMD-tuned scan, no encode/decode work).

The `Utf8Transcoder.cs` file remains in the repo but **fully commented out** — the class definition is preserved as historical reference / future reactivation if a workload ever surfaces where it could win again. `Utf8TranscoderTests.cs` is not currently exercising live code.

### Lesson — the V4N3 audit's blind spot

The V4N3 (custom transcoder) audit compared against **legacy `Encoding.UTF8.GetBytes`** and won. **The audit did NOT compare against `Utf8.FromUtf16`** (the modern API, .NET 7+). On modern runtime the BCL has two UTF-8 transcoders: a legacy one (instance-method on `Encoding`, virtual dispatch) and a modern one (static `Utf8.FromUtf16` / `Utf8.ToUtf16`). MemoryPack uses the modern one — that's what we should have been comparing against from the start.

**Generalizable lesson**: when measuring a custom implementation against a "BCL baseline", verify which BCL API is used by the actual competition (here: MemoryPack source-gen). The `Encoding.UTF8.*` instance API and `System.Text.Unicode.Utf8` static API are different generations of the same logical operation; treating them as interchangeable hides the comparison's scope.

### Why P1

- Closed the FastMode Compact mode Ser deficit from +20-29% to ≤11% on every cell (Latin1Long benchmark)
- Flipped the Deser side from -1 to -10% deficit to **AcBinary winning on 3 of 5 cells**, parity on 2 (Latin1Long benchmark)
- One-time fixed cost (8 production call-site cseréje) — every future bench profits
- Removed a load-bearing ~600-line custom SIMD module from the maintained surface area; future maintainers don't need to reason about Vector512BW / cross-lane shuffle / 5-popcount surrogate-pair correctness — the BCL handles it

### Follow-up — `Utf8Transcoder.cs` cleanup

The file is fully commented out. Either:
- **Delete** entirely (preferred for repo cleanliness) — `Utf8TranscoderTests.cs` then needs deletion or revival as a regression-only guard
- **Keep** the comment-block as historical reference, with a header comment pointing to this entry

Decision deferred — the comment-block does no harm to build / runtime. Address when the next docs-archive sweep runs.
