# AcBinary Wire Format

Complete wire format specification for the AcBinary serializer. Source of truth: `Serializers/Binaries/BinaryTypeCode.cs`.

> Features (interning, ref tracking, property ordering): `BINARY_FEATURES.md` | Options/presets: `BINARY_OPTIONS.md`
> Implementation (zero-alloc, buffer management): `BINARY_IMPLEMENTATION.md` | SGen architecture: `BINARY_SGEN.md`

## Stream Layout

```
[version : 1 byte]  [flags : 1 byte]  [cacheCount : VarUInt?]  [payload...]
```

- **version** — `FormatVersion = 1` (current).
- **flags** — See [Header Flags](#header-flags).
- **cacheCount** — Present only when `HeaderFlag_HasCacheCount` is set. Number of type wrapper slots used by serializer.

## Header Flags

The flags byte uses `0xB0` (176) as base with bit flags in the lower nibble. (Moved from `0x90` / 144 to make codepoints 135-167 contiguous for the FixStrAscii / StringAscii string-marker block.)

| Bit | Mask | Flag | Meaning |
|-----|------|------|---------|
| 0 | `0x01` | Metadata | Property hash metadata included (cross-type deserialization) |
| 1 | `0x02` | RefHandling_OnlyId | Reference tracking for `IId` objects only |
| 2 | `0x04` | RefHandling_All | Reference tracking for all objects (always combined with bit 1) |
| 3 | `0x08` | HasCacheCount | VarUInt cache count follows the flags byte |

**Reference handling modes:** None = `0x00`, OnlyId = `0x02`, All = `0x06` (bits 1+2).

## Variable-Length Encoding

### VarUInt (unsigned 32-bit)

Prefix-tier encoding (UTF-8-style): the first byte's high bits determine total size,
then the remaining bytes are read as a fixed-size little-endian value. No continuation-loop.

| First byte | Total | Bit-budget | Value range |
|---|---|---|---|
| `0xxxxxxx` | 1 byte | 7 | `0 .. 127` |
| `10xxxxxx` + 1B | 2 byte | 14 | `128 .. 16 383` |
| `110xxxxx` + 2B LE | 3 byte | 21 | `16 384 .. 2 097 151` |
| `1110xxxx` + 3B LE | 4 byte | 28 | `2 097 152 .. 268 435 455` |
| `1111xxxx` + 4B LE | 5 byte | 32 | `268 435 456 .. 4 294 967 295` (prefix nibble unused) |

Wire-size is identical to the legacy LEB128 encoding across all 5 tier-boundaries (7/14/21/28/32 bit) —
auto-inc IDs and other large values pay the same byte count. Decode wins: 1 if-ladder (max 4 branches,
JIT jump-table-friendly) + 1 fix-sized load on 3+ byte tiers, instead of N×ReadByte + N×shift +
N×continuation-check.

Encode (`WriteVarUIntMultiByteUnsafe`) uses `BitOperations.Log2` to pick the tier in O(1), then
packs the prefix byte + value bytes into a single `Unsafe.WriteUnaligned<uint>` store on the 2/3/4-byte
tiers (5-byte tier: one byte + one uint32 store).

### VarInt (signed 32-bit)

ZigZag encoding maps signed to unsigned, then VarUInt (prefix-tier above):

```
encode: (value << 1) ^ (value >> 31)
decode: (raw >> 1) ^ -(raw & 1)
```

Maps: `0 → 0`, `-1 → 1`, `1 → 2`, `-2 → 3`, etc.

### VarULong / VarLong (unsigned / signed 64-bit)

Legacy LEB128 encoding (7 data bits per byte, MSB = continuation flag). Max 10 bytes for uint64.
The 64-bit variants kept LEB128 — the prefix-tier benefit is concentrated in the 32-bit `VarUInt`
hot-path (auto-inc IDs, cache-indices, length prefixes, charLength offsets); 64-bit values are
rarely VarUInt-encoded.

## Type Markers

All markers defined in `BinaryTypeCode.cs`. `SlotCount = 64`.

### FixObj (0–63)

Single-byte object type. The marker byte **is** the type slot index — no additional type identifier needed.

```
[FixObj(N)]  [properties...]
```

**Slot allocation:** Slots 0–63 reserved for runtime polymorphic types, assigned dynamically on first encounter. Source-generated (SGen) types receive slots from 64+ via `AllocateWrapperSlot()` (sequential, `Interlocked.Increment`). SGen slots are compile-time stable; runtime slots depend on serialization order.

### Complex Types (64–71)

| Code | Name | Wire format |
|------|------|-------------|
| 64 | Object | `[64] [VarUInt typeIndex] [properties...]` |
| 65 | ObjectRef | `[65] [VarUInt refCacheIndex]` |
| 66 | Array | `[66] [VarUInt count] [elements...]` |
| 67 | Dictionary | `[67] [VarUInt count] [key, value pairs...]` |
| 68 | ByteArray | `[68] [VarUInt length] [raw bytes]` |
| 69 | ObjectWithMetadata | `[69] [VarUInt typeIndex] [VarUInt hashCount] [hashes...] [properties...]` |
| 70 | ObjectRefFirst | `[70] [VarUInt refCacheIndex] [object body...]` |
| 71 | ObjectWithMetadataRefFirst | `[71] [VarUInt refCacheIndex] [metadata + properties...]` |

### Polymorphic Types (72–75)

Used when runtime type differs from declared property type and `UseMetadata=false`.

| Code | Name | Wire format |
|------|------|-------------|
| 72 | ObjectWithTypeName | `[72] [UTF8 typeName] [inner marker] [body...]` — prefix, inner Object/Array/Dict follows |
| 73 | ObjectWithTypeNameRefFirst | `[73] [UTF8 typeName] [VarUInt refCacheIndex] [properties...]` — combined, no inner marker |
| 74 | ObjectWithTypeIndex | `[74] [VarUInt typeIndex] [inner marker] [body...]` — prefix |
| 75 | ObjectWithTypeIndexRefFirst | `[75] [VarUInt typeIndex] [VarUInt refCacheIndex] [properties...]` — combined |

Second occurrence of a referenced polymorphic object uses plain `ObjectRef(65)` — no polymorphic prefix needed.

### Primitives (76–90)

| Code | Name | Wire format |
|------|------|-------------|
| 76 | Null | `[76]` — no payload |
| 77 | True | `[77]` — no payload |
| 78 | False | `[78]` — no payload |
| 79 | Int8 | `[79] [1 byte]` |
| 80 | UInt8 | `[80] [1 byte]` |
| 81 | Int16 | `[81] [VarInt]` |
| 82 | UInt16 | `[82] [VarUInt]` |
| 83 | Int32 | `[83] [VarInt]` |
| 84 | UInt32 | `[84] [VarUInt]` |
| 85 | Int64 | `[85] [VarLong]` |
| 86 | UInt64 | `[86] [VarULong]` |
| 87 | Float32 | `[87] [4 bytes IEEE 754]` |
| 88 | Float64 | `[88] [8 bytes IEEE 754]` |
| 89 | Decimal | `[89] [16 bytes]` |
| 90 | Char | `[90] [VarUInt]` |

### Strings (91, 92, 93, 104, 105, 167)

| Code | Name | Wire format |
|------|------|-------------|
| 91 | StringUtf16 | `[91] [charLen:int32 LE] [UTF-16 raw bytes]` — FastWire mode UTF-16 payload (no UTF-8 transcoding; speed > size) |
| 92 | StringInterned | `[92] [VarUInt cacheIndex]` — 2nd+ occurrence of an interned string |
| 93 | StringEmpty | `[93]` — empty string, no payload |
| 104 | StringInternFirstSmall | `[104] [VarUInt cacheIdx] [charLen:8][utf8Len:8] [UTF-8 bytes]` — 1st occurrence interning tier, packed dual-length (utf8Len ≤ 255) |
| 105 | StringInternFirstMedium | `[105] [VarUInt cacheIdx] [charLen:16 LE][utf8Len:16 LE] [UTF-8 bytes]` — 1st occurrence interning tier, packed dual-length (utf8Len ≤ 65535) |
| 167 | String | `[167] [VarUInt(charLength - FixStrCount)] [unsigned excess:1\|2\|4] [UTF-8 bytes]` — universal long-form (charLength > FixStrMaxLength) |

The 167 `String` marker is the universal long-form string. It unifies the former `StringLen8/16/32`
(at 167/168/169) into a single marker with prefix-tier VarUInt charLength encoding (offset by
`FixStrCount = FixStrMaxLength + 1 = 32`, since FixStr already covers 0..31). The excess slot
width (1 / 2 / 4 bytes) is selected from charLength via `GetUniversalStringExcessSlotSize` and
encodes `bytesWritten - charLength` (= UTF-8 byte excess over UTF-16 char count) for the reader's
`string.Create(charLen) + Utf8.ToUtf16` fast path (avoids the pre-scan `Encoding.UTF8.GetCharCount` pass).

The interning tiers (`StringInternFirstSmall` / `Medium` at 104/105) keep their packed dual-length
(charLen + utf8Len) format — the post-encode tier choice exploits the typical interning workload
(short property names, enum strings) for fast deserialize via single packed-read.

ASCII strings flow through the same `String` / FixStr markers — the reader uses `excess == 0` as
the ASCII-validity discriminator (every UTF-16 char < 0x80 produces exactly 1 UTF-8 byte; non-ASCII
chars always produce 2-4 bytes). On excess=0 the reader takes the byte→char widen fast path
(`Encoding.Latin1.GetString` SIMD); on excess>0 the UTF-8 decode runs.

Slots 168, 169 are **Reserved** — freed by the StringLen8/16/32 unification, available for future
marker allocation (see Reserved Ranges below).

### Date/Time (95–98)

| Code | Name | Wire format |
|------|------|-------------|
| 95 | DateTime | `[95] [8 bytes ticks]` |
| 96 | DateTimeOffset | `[96] [8 bytes ticks] [VarInt offsetMinutes]` |
| 97 | TimeSpan | `[97] [VarLong ticks]` |
| 98 | Guid | `[98] [16 bytes]` |

### Other Markers

| Code | Name | Wire format |
|------|------|-------------|
| 99 | Enum | `[99] [VarInt underlyingValue]` |
| 100 | MetadataHeader | Legacy: implies `RefHandling=true` + metadata present |
| 101 | NoMetadataHeader | Legacy: implies `RefHandling=true`, no metadata |
| 102 | PropertySkip | `[102]` — marks skipped property (default/null value) |

### FixStr (135–166) — short universal string marker

Short strings encoded in a single marker byte + raw UTF-8 bytes (no length prefix — charLength
encoded in the marker offset). The H2Q6 reorganization merged the formerly-split FixStr (UTF-8 at
103-134) and FixStrAscii (135-166) into one universal FixStr block at 135-166; codepoints 103..134
are now part of the Reserved Range (see below).

```
[FixStrBase + charLength]  [UTF-8 bytes]
```

- Length range: 0..31 chars (`FixStrBase = 135`, `FixStrMax = 166`, `FixStrMaxLength = 31`)
- `FixStrCount = FixStrMaxLength + 1 = 32` — single source of truth for the FixStr slot count,
  also the wire-format offset for the long-form `String` marker's VarUInt charLength (`wireLen =
  charLength - FixStrCount`). If the FixStr range ever expands (e.g. 32 → 64 slots), this constant
  updates the offset everywhere consistently.
- Saves header bytes vs `String` marker + VarUInt length (1 marker byte total vs 3+ byte header)
- Content semantics: UTF-8 (may contain multi-byte sequences for non-ASCII chars)
- ASCII discriminator: the reader uses the post-decode `excess == 0` check; ASCII-only strings
  bypass the UTF-8 decode via `Encoding.Latin1.GetString` SIMD byte→char widen.

### Reserved ranges

- **103..134** (29 slots): freed by H2Q6 FixStr unification (formerly the non-ASCII FixStr range).
  Active reservations per `BINARY_TODO.md` marker-tier reorganization plan:
  - 104: `StringInternFirstSmall` (active)
  - 105: `StringInternFirstMedium` (active)
  - 106..134: reserved for `ACCORE-BIN-T-L9Y3` FixArray short-list count (16 values), `S5L8`
    sentinel-length tiers (5 values), `S2X9` markerless schema lane (4 values), `F3W6` dedicated
    FastWire string marker (1 value), general reserve (3 values)
- **168..169** (2 slots): freed by `String` marker unification (formerly StringLen16, StringLen32).
  Available for the upcoming `FixStr` range expansion (32→64 chars) — would extend FixStr from
  `135..166` to `135..198`, absorbing 168/169 and pushing the next free slot to 199.
- **170..175** (6 slots): pre-existing reserve for future string-related markers (e.g., compressed
  / base64 / mixed-ASCII variants), keeping the 91..167 range a single contiguous string-marker block.

### TinyInt (192–255)

Single-byte integer encoding for small values:

```
value = marker - 192 - 16    (range: -16 to 47)
marker = value + 16 + 192    (64 values total)
```

Saves 2+ bytes vs `Int32(83)` + VarInt for frequently occurring small integers.
