AyCode.Core/AyCode.Core/docs/BINARY/BINARY_FEATURES.md

AcBinary Features

Advanced serialization features on top of the wire format. Wire format: BINARY_FORMAT.md | Options/presets: BINARY_OPTIONS.md | Internal architecture: BINARY_IMPLEMENTATION.md | Source generation: BINARY_SGEN.md.

Architectural framing — why use AcBinary at all? See BINARY_WHYUSE.md for the category positioning vs wire-only serializers (Protobuf / MessagePack / MemoryPack), the three-pillar value proposition (graph integrity + bandwidth + streaming), real-world WASM SignalR reference numbers, fit/not-fit decision lists, and the framing for how to read AcBinary's benchmark numbers.

Optimization Policy (LLM)

AcBinary is a general-purpose serializer, not a benchmark-only implementation.

When proposing or implementing performance work, optimize for broad real-world workloads and maintain balanced trade-offs across:

• mixed payload shapes (small/medium/large/deep)
• language distributions (ASCII-heavy, mixed Latin, multi-byte UTF-8 such as CJK)
• throughput, latency, allocation, and wire size

Do not accept a change solely because one benchmark cell improves. Any optimization should be validated across multiple representative scenarios and must avoid benchmark-specific overfitting.

Compact Encoding Selection

The serializer applies compact encodings automatically:

Data	Condition	Encoding	Savings
Integer	−16 ≤ v ≤ 47	TinyInt (1 byte)	2–5 bytes
String	≤31 bytes UTF-8, any content	FixStr (1+N bytes)	1 byte (no length prefix)
String	≤31 bytes, pure ASCII	FixStrAscii (1+N bytes)	1 byte + reader skips UTF-8 decode
String	>31 bytes, pure ASCII	StringAscii (1+VarUInt+N bytes)	reader skips UTF-8 decode
Object	type index < 64	FixObj (1 byte)	1–5 bytes (no VarUInt index)
String	empty	StringEmpty (1 byte)	1+ bytes
Bool	—	True/False (1 byte)	no payload

ASCII marker-dispatch

The writer's WriteStringWithDispatch runs a single-pass UTF-8 encode and detects pure-ASCII content for free via bytesWritten == charLength (every UTF-16 char < 0x80 produces exactly 1 UTF-8 byte; non-ASCII chars always produce 2-4 bytes). Based on the result it emits one of four markers:

• FixStrAscii (135-166) — short ASCII (≤31 bytes)
• FixStr (103-134) — short UTF-8 (≤31 bytes, mixed/multi-byte content)
• StringAscii (167) — long ASCII (>31 bytes)
• String (91) — long UTF-8 (>31 bytes, mixed/multi-byte content)

The reader uses the marker as the ASCII-validity contract — FixStrAscii / StringAscii payloads byte→char widen directly via Encoding.Latin1.GetString (BCL SIMD-accelerated, ~memcpy class throughput), no UTF-8 decode, no run-time Ascii.IsValid scan. FixStr / String payloads use the custom 3-phase UTF-8 decoder (Vector256 ASCII prefix widen + DWORD ASCII batch + scalar multi-byte branch). Wire format unchanged across format versions — the new markers occupy previously-unused codepoints, so wire produced without ASCII detection (older writers) is forward-compatible.

String Interning Protocol

Controls deduplication of repeated string values.

Modes (StringInterningMode):

• None — all strings inline, no overhead
• Attribute — only [AcStringIntern] properties interned (default)
• All — all strings within length limits interned

Length limits: MinStringInternLength=4, MaxStringInternLength=64 (configurable).

Wire protocol:

1. Serializer pre-scans all eligible strings to build a plan (which strings repeat)
2. First occurrence: [StringInternFirst(94)] [VarUInt cacheIndex] [VarUInt byteLength] [UTF-8 bytes]
3. Subsequent: [StringInterned(92)] [VarUInt cacheIndex]
4. Single-occurrence strings: written as normal String/FixStr (no interning overhead)

Reference Tracking

Prevents infinite loops and preserves object identity for repeated references.

Modes (ReferenceHandlingMode):

• None — no tracking (fastest, use when graph is a tree)
• OnlyId — track only IId objects (matched by ID value)
• All — track all reference types (two-phase scan required)

Two-phase process:

1. Scan pass (ScanPass.cs) — walks the object graph, detects multi-referenced objects and repeated strings. Builds a WriteDuplicateEntry[] array (the "write plan") containing VisitIndex, CacheMapIndex, IsFirst, and Value for each duplicate.
2. Sort — write plan entries are sorted by VisitIndex to match the write pass traversal order.
3. Serialize pass — consumes the sorted write plan via TryConsumeWritePlanEntry(). A cursor (_nextWritePlanVisitIndex) advances through the plan in O(1) — no dictionary lookups during serialization.

Wire protocol:

• First occurrence: [ObjectRefFirst(70)] [VarUInt refCacheIndex] [object body...]
• Subsequent: [ObjectRef(65)] [VarUInt refCacheIndex]

Example — same object referenced twice:

Input:  { Users: [userA, userA] }   (same instance)

Scan pass → WritePlan:
  [{VisitIndex:2, CacheMapIndex:0, IsFirst:true},
   {VisitIndex:3, CacheMapIndex:0, IsFirst:false}]

Wire output (Compact mode, ReferenceHandling=All):
  [version=1] [flags=0x96]  [VarUInt cacheCount=1]     ← header
  [FixObj(0)]                                           ← root object
    [Array(66)] [VarUInt(2)]                            ← Users array, 2 elements
      [ObjectRefFirst(70)] [VarUInt(0)] [props...]      ← userA, 1st occurrence
      [ObjectRef(65)] [VarUInt(0)]                      ← userA, 2nd (2 bytes only)

Hybrid Execution Model (Runtime vs Source Generated)

Two execution modes, seamlessly interoperable in a single serialization run:

Mode	Trigger	Property access	When to use
SGen	[AcBinarySerializable] + UseGeneratedCode=true	Unsafe.As<T> direct	Hot-path types
Runtime	No attribute or UseGeneratedCode=false	Compiled delegates	3rd-party types, fallback

SGen root types use a fast path that skips the full dispatch chain (~12 calls → 3 checks). SGen children call directly into other SGen writers; non-SGen children fall back to runtime via bridge methods. Wire format is identical regardless of mode.

Full SGen architecture, bridge methods, generated code patterns, wrapper slots: BINARY_SGEN.md

NativeAOT Compatibility

Both execution modes work under NativeAOT publish. SGen is the recommended path; Runtime works but is significantly slower.

Path	AOT compatible?	Performance vs JIT	Recommendation
SGen ([AcBinarySerializable] + UseGeneratedCode=true)	✅ Full	Same as JIT	Production AOT default
Runtime (no attribute or UseGeneratedCode=false)	✅ Functional	~5-7x slower than SGen	Fallback for unattributed types

Two-axis AOT strategy

NativeAOT has two distinct constraints — both must be addressed for the Runtime path to work:

1. Dynamic code generation prohibited. Expression.Compile() and Reflection.Emit fail under AOT (no JIT engine in the binary). The 7 property-accessor + constructor factories in AcSerializerCommon use if (!RuntimeFeature.IsDynamicCodeSupported) runtime guards to fall back to plain reflection delegates (PropertyInfo.GetValue/SetValue, ConstructorInfo.Invoke).

2. Reflection metadata trim. The trimmer drops Type.GetConstructor() / Type.GetProperties() metadata for types not statically referenced. The library propagates [DynamicallyAccessedMembers(PublicParameterlessConstructor | PublicProperties)] from the public Deserialize<T> / Serialize<T> entry points down through the metadata classes. The RequiredMembers constant on TypeMetadataBase is the single source of truth for the DAMs requirement.

Documented trimmer blind spots

Two well-known limitations require consumer cooperation:

• Polymorphism via obj.GetType() — WritePropertyOrSkip writes the runtime concrete type for polymorphic properties; the trimmer cannot follow GetType() flow. Consumer must root polymorphic concrete types: either via [AcBinarySerializable] on each (SGen path covers them) or <TrimmerRootAssembly> for the data-model assembly. The IL2072 warning is suppressed at this site with documented justification.
• Nested types via property-type chain — List<TestOrderItem> element type extracted via Type.GetGenericArguments() loses DAMs context. Same mitigation: [AcBinarySerializable] on each nested type, or <TrimmerRootAssembly>. The IL2072/IL2075 warnings are suppressed at the relevant sites with documented justification.

Consumer guidance for AOT publish

• Annotate every type in the serialization graph with [AcBinarySerializable] — SGen path is AOT-clean and ~5-7x faster than the Runtime fallback.
• Or, for unattributed types: add <TrimmerRootAssembly Include="MyDataModel" /> in the consumer's csproj to keep reflection metadata for the data-model assembly intact (cost: larger AOT binary).
• The MessagePackSerializer.Standard resolver has the same Reflection.Emit problem — use MessagePack source generators or skip MessagePack under AOT (the AcBinary benchmark project does the latter via #if !AYCODE_NATIVEAOT).
• System.Text.Json reflection mode similarly requires JsonSerializerIsReflectionEnabledByDefault=true AND consumer awareness; for AOT the source-gen path is preferred.

Verified

The full benchmark suite (Small/Medium/Large/Repeated Strings/Deep Nested) runs cleanly under NativeAOT publish on Windows x64 + .NET 9. AcBinary SGen retains its +12-35% speed and -22-33% wire-size advantage over MemoryPack under AOT, identical to JIT. Runtime path is functional but ~5-7x slower (reflection-delegate overhead).

Property Ordering

Properties are serialized in a deterministic order defined by TypeMetadataBase.GetUnfilteredProperties():

1. Walk the inheritance chain from derived → base (currentType.BaseType loop)
2. At each level, collect declared public instance properties
3. Sort alphabetically (StringComparer.Ordinal) within each level
4. Result: base properties first, then derived, alphabetical within each level

This order is stable across serializer/deserializer as long as the type hierarchy doesn't change.

Cross-Type Deserialization (UseMetadata)

When UseMetadata=true, property name hashes (FNV-1a via FnvHash.ComputeString) are written per type, enabling schema evolution:

• Serializer writes property hashes in the metadata section (ObjectWithMetadata(69))
• Deserializer builds an index mapping array (GetIndexMapping()) — maps source property indices to destination via FNV-1a hash matching
• Allows deserialization across different property sets or ordering

When UseMetadata=false, properties are matched by positional index only — source and destination must have identical property layouts.

Edge cases:

• Hash collision (CheckDuplicatePropName=true, default): throws InvalidOperationException. ⚠️ When false: collision silently ignored — risk of data corruption, see BINARY_ISSUES.md#accore-bin-i-c5r7.
• Source has unknown property (not in destination): silently skipped via SkipValue(), no error.
• Destination has extra property (not in source): left at default value (new instance) or unchanged (populate mode).