AyCode.Core/AyCode.Core/docs/BINARY/BINARY_ISSUES.md

Binary Serializer — Known Issues & Limitations

This page covers issues in the binary serializer core (format, SGen, options, deserialization context, buffer writer). Issues specific to the streaming I/O layer (AsyncPipeReaderInput + AsyncPipeWriterOutput, multi-message wire framing, sliding-window buffer, producer-consumer synchronization) are tracked separately in BINARY_ASYNCPIPE_ISSUES.md.

Deserialization

ACCORE-BIN-I-D2J5: Non-array-backed memory — per-segment copy

Status: Open Affects: SequenceBinaryInput Path: ExtractArray() fallback when MemoryMarshal.TryGetArray fails

When ReadOnlySequence<byte> segments are backed by native memory (not managed byte[]), each segment is copied into a new byte[]. This is unavoidable — the context requires byte[] for Unsafe.ReadUnaligned, AsSpan, and Encoding.GetString.

Impact: Negligible. Non-array-backed ReadOnlyMemory is extremely rare (custom MemoryManager<T> with native memory, memory-mapped files). All standard .NET pools (ArrayPool, MemoryPool.Shared, Kestrel pipe) are array-backed.

ACCORE-BIN-I-G7N3: Cross-boundary scratch buffer is not pooled across calls

Status: Open Affects: SequenceBinaryInput._scratchBuffer

The scratch buffer is ArrayPool.Rent-ed on first cross-boundary read and reused within a single deserialization. It is Return-ed in Release() after deserialization completes. However, the next deserialization will rent again.

Impact: Minimal. ArrayPool.Shared reuses buffers efficiently. The scratch is typically small (4-16 bytes for fixed-width boundary reads). Large scratch (>4KB) only occurs when a string or byte[] straddles a segment boundary.

Possible optimization: Store the scratch buffer on the pooled BinaryDeserializationContext and reuse across deserializations. Low priority — ArrayPool overhead is negligible.

ACCORE-BIN-I-S1F8: ReadBytes always copies

Status: Open Affects: BinaryDeserializationContext.ReadBytes(int length)

ReadBytes allocates a new byte[] and copies from the buffer. This is unavoidable because the caller owns the returned array, and the source buffer (pipe segment or serialized data) may be recycled.

ACCORE-BIN-I-V5L2: ReadStringUtf8 requires contiguous buffer

Status: Open Affects: BinaryDeserializationContext.ReadStringUtf8(int length)

Encoding.GetString and Ascii.IsValid require contiguous memory. For multi-segment reads, EnsureAvailable copies cross-boundary bytes into the scratch buffer first. This is the same approach SequenceReader<byte> uses internally.

Possible optimization: Span-by-span UTF-8 decode for cross-boundary strings (like MessagePack). Low priority — most strings are shorter than a segment (4KB).

Serialization

ACCORE-BIN-I-K8R4: BufferWriterBinaryOutput fallback path allocates per-chunk

Status: Open Affects: BufferWriterBinaryOutput.AcquireChunk fallback

When MemoryMarshal.TryGetArray fails on IBufferWriter.GetMemory() (native memory-backed writer), a byte[] is rented from ArrayPool per chunk and copied to the writer on Grow/Flush. Same as ACCORE-BIN-I-D2J5 — non-array-backed writers are extremely rare.

ACCORE-BIN-I-P3M6: AsyncPipeWriterOutput uses sync GetResult() for backpressure

Status: Open Affects: AsyncPipeWriterOutput.Grow() — _lastFlush.GetAwaiter().GetResult()

When the previous PipeWriter.FlushAsync() hasn't completed by the next Grow() call, the serializer blocks the thread until the flush completes. This is necessary because IHubProtocol.WriteMessage is void (synchronous by design).

Impact: Minimal under normal conditions. PipeWriter.FlushAsync() writes to an in-memory Kestrel pipe (not directly to the network) and typically completes synchronously. Only blocks when the pipe's internal buffer hits its pause threshold (~1MB), which requires an extremely slow client + large payload. The Bytes mode (default) has the same blocking characteristic — it blocks the thread for the entire serialization + single flush.

Possible optimization: AsyncSegment mode (future) with a custom async WriteMessageAsync protocol interface, enabling await on flush instead of GetResult().

ACCORE-BIN-I-T9X1: AsyncPipeWriterOutput fallback path — same as ACCORE-BIN-I-K8R4

Status: Open Affects: AsyncPipeWriterOutput.AcquireChunk fallback

Same TryGetArray fallback as BufferWriterBinaryOutput (ACCORE-BIN-I-K8R4). Kestrel PipeWriter.GetMemory() always returns array-backed memory — fallback is for non-standard PipeWriter implementations only.

ACCORE-BIN-I-K3W8: FastWire skips string interning → shared write-plan cursor desyncs ref handling

Status: Open · Severity: Major (latent — silent data corruption) Affects: WireMode.Fast + a type with BOTH [AcStringIntern] properties AND reference handling.

The FastWire string path (WriteStringGenerated / WritePropertyOrSkip String case) early-returns via WriteStringUtf16Markerless and never reaches WriteString's interning block → string interning is skipped in FastWire. But the scan pass is FastWire-agnostic and still emits string-intern entries into the write plan, which is shared (VisitIndex-ordered, single cursor) with reference handling. FastWire never consumes its string entries → the shared cursor (TryConsumeWritePlanEntry) desyncs → ref handling reads the wrong entry → shared objects duplicated / object identity lost. Compact mode is fine → latent (the benchmark uses Compact, so it has gone undetected).

Fix direction: FastWire must be purely a string byte-encoding choice (UTF-16 raw vs UTF-8 markered) — interning + ref handling must behave identically to Compact. Route FastWire string writes through the interning protocol (InternFirst/InternRef markers, cache, write-plan consumption); only the first-occurrence content bytes differ. Drop the early-return. Conceptually this is the long-proposed WireMode → StringEncoding rename.

Note: the cursor-desync mechanism is strongly inferred (FastWire-gate grep + the shared write-plan design) — confirm against TryConsumeWritePlanEntry + the scan pass before implementing the fix.

Deserialization (PipeReader)

ACCORE-BIN-I-B4Y7: PipeReaderBinaryInput uses sync ReadAsync().GetResult()

Status: Open Affects: PipeReaderBinaryInput.Initialize() and TryAdvanceSegment()

Same constraint as ACCORE-BIN-I-P3M6 — IBinaryInputBase interface is synchronous. ReadAsync().GetAwaiter().GetResult() blocks when waiting for more data from the pipe. Currently not used in production (SignalR delivers complete messages via TryParseMessage). Reserved for future direct-pipe deserialization scenarios.

Source Generator (SGen)

ACCORE-BIN-I-H2C5: CS8625 warnings for non-nullable reference types

Status: Open Affects: Generated reader code

The source generator emits null assignments for non-nullable reference type properties during deserialization (before the value is read from the stream). This produces CS8625 warnings. Functionally harmless — the property is always assigned before use.

ACCORE-BIN-I-N6Q3: First-run cold-start overhead

Status: Open Affects: First Serialize<T>/Deserialize<T> per [AcBinarySerializable] type, per process

Cold-start cost chain on first use of an SGen type (before ACCORE-BIN-T-W9F1 lands):

1. BinarySerializeTypeMetadata ctor — reflection property enumeration + GetCustomAttribute scans
2. Expression.Compile per property accessor (dynamic getter + typed getters) — dominant cost
3. TypeMetadataWrapper ctor — GeneratedWriterRegistry + GeneratedReaderRegistry lookups, tracking state init
4. JIT of WriteObject / WriteObjectProperties / scan pass
5. JIT of generated WriteProperties / ScanObject / ScanForDuplicates (size scales with property count)
6. Cascade: each referenced child type repeats steps 1–5

Subsequent calls hit cached metadata/wrappers → only Tier 0→1 JIT transition remains (background, async).

Dominant cost today: #1–#2 (reflection + Expression.Compile). After ACCORE-BIN-T-W9F1, the dominant residual cost shifts to #4–#5 (JIT), addressed by ACCORE-BIN-T-T5J8.

Impact: Measurable first-call latency — larger for types with many properties or deep graphs. For SignalR workloads the first message per entity type pays this tax.

ACCORE-BIN-I-F1W8: Consumer entity with new Id shadowing — excluded from SGen

Status: Open Affects: Any consumer entity whose base class hides BaseEntity.Id with readonly new int Id { get; } pattern (e.g. DiscountProductMapping in Mango.Nop.Core)

When the base class shadows Id with a setter-less new int Id { get; }, SGen can't emit a setter without CS0200. Runtime falls back to compiled-expression serialization for these types. Low priority — affects a small number of consumer entities.

Related TODO: BINARY_TODO.md#accore-bin-t-q2n7

Buffer Writer (BWO)

ACCORE-BIN-I-J4D2: Struct copy semantics

Status: Open Affects: BufferWriterBinaryOutput value-type assignment

Assigning a BufferWriterBinaryOutput value creates an independent copy. State changes (e.g. _committedBytes via Grow/Flush) are not reflected in the original. Copy back after use if needed.

ACCORE-BIN-I-R5V9: Initialize resets tracking

Status: Open Affects: BufferWriterBinaryOutput.Initialize (context mode)

Initialize sets _committedBytes = 0. Standalone bytes written before are lost if the BWO is then passed to a context. Call FlushAndReset() first, or track standalone bytes separately.

ACCORE-BIN-I-L7G3: Constructor acquires chunk

Status: Open Affects: BufferWriterBinaryOutput ctor

AcquireChunk runs in ctor for standalone readiness. Redundant if only context mode is used (context Initialize acquires its own). Not a leak — consecutive GetMemory without Advance returns overlapping memory.

ACCORE-BIN-I-M3K6: No mode mixing

Status: Open Affects: BufferWriterBinaryOutput — context vs standalone mode

A single instance must not use context + standalone modes simultaneously — buffer states desynchronize. One mode per lifecycle phase; FlushAndReset() as boundary between modes.

ACCORE-BIN-I-Q4P7: ArrayBinaryOutput.DetachResult ownership transfer missing → pooled buffer double-return

Status: Open · Severity: Critical (latent — silent cross-talk corruption) Affects: ArrayBinaryOutput.DetachResult, ArrayBinaryOutput.Reset, ArrayBinaryOutput.Dispose

DetachResult returns new BinarySerializationResult(resultBuffer, ..., pooled: true), which transfers buffer ownership to the result object (caller disposes result → buffer returned to ArrayPool). But _rentedBuffer keeps referencing the same array after detach. Later Reset (large-buffer branch) or Dispose returns _rentedBuffer again, causing a double-return of the same array to ArrayPool.Shared.

Impact: Silent, intermittent data corruption. ArrayPool can hand out the same physical array to multiple renters after double-return, enabling cross-talk between unrelated serialization operations.

Why this is active in default config: ctor default initialCapacity=65535, while MaxKeepBufferSize=32KB; the detached default buffer is considered "large", so Reset naturally enters the return-to-pool path.

Fix direction: Treat DetachResult as a strict ownership transfer boundary. After detach, _rentedBuffer must no longer point to the detached array. Possible implementation variants:

• Eager replacement: rent a replacement buffer immediately in DetachResult.
• Lazy replacement: set _rentedBuffer = null in DetachResult, and perform lazy-rent in Initialize (_rentedBuffer ??= ArrayPool<byte>.Shared.Rent(_initialCapacity)).

Lazy replacement avoids redundant rent/return churn when DetachResult is followed by Reset or Dispose, while preserving single-owner semantics (the detached BinarySerializationResult remains the only owner until it calls Return in Dispose). Ensure exactly one return path per rented array.

ACCORE-BIN-I-R2D6: ArrayBinaryOutput.Reset may set _rentedBuffer to null while Initialize assumes non-null

Status: Open Affects: ArrayBinaryOutput.Reset, ArrayBinaryOutput.Initialize

Reset currently does:

_rentedBuffer = nextCapacity == _initialCapacity ? null : ArrayPool<byte>.Shared.Rent(nextCapacity);

So _rentedBuffer can become null after returning a large buffer, while Initialize unconditionally reads _rentedBuffer and _rentedBuffer.Length.

Impact: Deterministic NullReferenceException on the next initialization path when the null branch is taken.

Fix direction: Keep _rentedBuffer always non-null by renting _initialCapacity in that branch, or add lazy-rent null handling in Initialize before dereference.

ACCORE-BIN-I-V8N4: BinarySerializationResult accessors remain usable after Dispose → pooled-buffer use-after-return

Status: Open · Severity: Critical (latent — silent data corruption) Affects: BinarySerializationResult.Buffer, BinarySerializationResult.Span, BinarySerializationResult.Memory, BinarySerializationResult.Dispose

BinarySerializationResult.Dispose returns the underlying array to ArrayPool when pooled=true, but public accessors (Buffer / Span / Memory) remain callable without a disposed guard. After dispose, the same array may already be re-rented and mutated by unrelated operations; reading the old result then becomes use-after-return on pooled memory.

Impact: Silent, non-deterministic cross-talk corruption. Consumers may observe stale/foreign bytes through Span / Memory / Buffer with no exception signal.

Possible fix directions:

• Add _disposed guard to all accessors (ObjectDisposedException after dispose).
• Optionally scrub/neutralize post-dispose state (e.g., replace exposed buffer reference with Array.Empty<byte>()) to reduce accidental reuse risk.
• Clarify API ownership contract in docs: disposed result is terminal and must not be accessed.

Configuration / Options

ACCORE-BIN-I-L8N5: AcBinarySerializerOptions thread-safety — mutable properties on shared instances

Status: Open Affects: AcBinarySerializerOptions — all set; properties (UseMetadata, UseGeneratedCode, WireMode, UseStringInterning, BufferWriterChunkSize, UseCompression) and any holder that retains a shared instance (e.g. DI-scoped serializer wrapper, long-lived service field, AcBinaryHubProtocol._options).

The options class exposes mutable properties. When a consumer shares one options instance across concurrent Serialize / Deserialize calls — common with DI-singleton services, hot-path-cached options, or any long-lived holder — a runtime property change is observable mid-operation by other in-flight calls. Result: invariant violations, mismatched encoding decisions, intermittent output / deserialization corruption.

Worsened by AcBinaryHubProtocol ctor (AyCode.Services/SignalRs/AcBinaryHubProtocol.cs sor 141), which mutates the caller-provided options reference (_options.BufferWriterChunkSize = options.BufferSize;) — the caller's external reference becomes a side-channel for the protocol's internal config.

A volatile field on the holder side (e.g. _options) only protects reference replacement, not property-level mutation; an external reference still in scope can be serOpts.UseGeneratedCode = false; mid-parse on another thread.

Impact: Latent — corruption is intermittent and timing-dependent; very hard to reproduce without targeted stress. The NuGet contract worsens this: the package cannot constrain how consumers scope their options instances.

Possible fix directions:

• Defensive copy on ingress — Clone() on AcBinarySerializerOptions; every API that retains an options instance clones it on entry. External mutation becomes invisible to the holder.
• Immutable record refactor — set; → init; on all configuration properties; mutation requires with-expression which produces a new instance.
• Read-only flag pattern — à la JsonSerializerOptions.MakeReadOnly(). The holder calls MakeReadOnly() on entry; subsequent mutation throws.

ACCORE-BIN-I-C5R7: CheckDuplicatePropName=false silently corrupts on FNV-1a hash collision

Status: Open Affects: AcBinarySerializerOptions.CheckDuplicatePropName when set to false

The default value (true) throws InvalidOperationException on FNV-1a property-name hash collision within a type. When set to false (a doc-suggested production-performance optimization), collisions are silently accepted — the second property's hash overwrites the first in the lookup table, and the wrong property setter is invoked during deserialization. Result: silent data corruption between the colliding properties.

Impact: Latent — FNV-1a + typical property names rarely collide, but applications with many SGen types eventually hit one. Detection requires a separate property-by-property comparison after round-trip; the serializer surfaces no signal.

Possible fix directions:

• Doc harmonization — single decision rule ("always true; perf cost is negligible vs. corruption risk").
• Wider hash — replace FNV-1a with xxHash3-128 (or similar) for collision-free property identification.
• Disambiguate by index — store property index alongside hash so a collision is detectable at deserialization without throwing on serialization.

ACCORE-BIN-I-P2H8: MaxDepth cut-off Null indistinguishable from real null

Status: Closed (2026-05-14) — reframed as explicit opt-in feature via MaxDepthBehavior enum on AcSerializerOptions. Affects: AcBinarySerializerOptions.MaxDepth (and any preset using a non-default value)

When the object graph exceeded MaxDepth, deeper objects/collections were written as Null(76) — the same byte as a genuine null value — without any opt-in. The deserializer could not distinguish "depth-cut-off null" from "real null", so unintentional truncation appeared as silent data loss.

Impact (historical): The unconditional silent truncation was the actual problem — developers couldn't see when it fired. The wire ambiguity itself is fine when the developer opts in (typical shallow-serialization use case: partial-update endpoint where "null nested collection" means "no change at this level" per the protocol contract).

Resolution

The shallow-serialization use case is a legitimate, common pattern (client edits a grid value → server gets a flat entity for DB update, with no need to round-trip nested sub-lists). The fix is explicit opt-in so unintentional truncation can't sneak through. AcSerializerOptions now exposes MaxDepthBehavior (enum, default Throw):

• Throw (new default) — InvalidOperationException with type name + position. Unintended depth-exceeded cases surface as a debuggable exception.
• Truncate — the previous WriteByte(Null) behavior, now explicit opt-in for shallow serialization (delta updates, view-model projections, partial DB-update flows). The wire Null at the truncation boundary is the developer's contract decision — endpoint protocol dictates what nested null means. Works with any persistence layer (Dapper, ADO.NET, Cosmos DB, MongoDB, Redis, EF Core, etc.).
• Disable — skip the depth check entirely (max perf, dev guarantees cycle-free graph).

The check moved from "every object/collection write" (with rewind) to "before any marker byte is written" (in WriteObject runtime + WriteObjectFullMarker* SGen). The FlatCopy preset was updated to explicitly set MaxDepthBehavior = Truncate to preserve its original "root + Null nested" semantic. See BINARY_OPTIONS.md MaxDepth + MaxDepthBehavior section for full details.

ACCORE-BIN-I-W3F4: PropertyFilter + UseMetadata=false silently corrupts via index drift

Status: Open Affects: AcBinarySerializerOptions.PropertyFilter combined with UseMetadata=false

When the serializer applies a PropertyFilter, excluded properties are completely absent from the stream — no marker, no placeholder. The deserializer must apply an identical filter, OR rely on UseMetadata=true property-name hash matching. If neither condition holds, positional indices on the receive side mis-match: property A's value lands in property B's setter → silent data corruption.

Impact: Severe in NuGet contexts — the package cannot enforce symmetric filter configuration on both ends. A common pattern ("send-side filter to drop sensitive fields") silently corrupts cross-deployment if the receiver isn't aware to mirror.

Possible fix directions:

• Emit PropertySkip(102) marker for filtered slots — the marker already exists on the wire, verify the write path uses it for filtered properties.
• Auto-promote to UseMetadata=true when PropertyFilter is set (with a warning) — opt-out via explicit override.
• Validate at serialize entry — PropertyFilter != null && !UseMetadata → throw InvalidOperationException with guidance.

ACCORE-BIN-I-J6T9: Non-IId circular references silently truncated when ThrowOnCircularReference=false

Status: Partially Fixed (2026-05-14) — default behavior now surfaces the cycle as an exception. Affects: AcBinarySerializerOptions.ThrowOnCircularReference=false combined with ReferenceHandling != None

With ThrowOnCircularReference=false + reference handling enabled, only IId-implementing types are tracked for cycle detection. Non-IId circular references hit MaxDepth before being detected → silent truncation at the depth boundary, no exception, no log.

Impact (historical): Borderline — explicit opt-in (developer must set ThrowOnCircularReference=false), and most domain models avoid non-IId cycles. But UI tree models (parent ↔ children with no Id), graph data structures, and self-referencing config nodes triggered this path silently.

Resolution (partial)

With the introduction of MaxDepthBehavior (default Throw, see ACCORE-BIN-I-P2H8), non-IId circular references now throw at the depth boundary instead of silently truncating — the safety-net global recursion counter (RecursionDepth byte field on the context, gated by _needsDepthCheck = !HasAllRefHandling && MaxDepthBehavior != Disable) fires at the MaxDepth limit and surfaces the cycle as InvalidOperationException with type name + position.

Remaining gap: The detection mechanism is depth-based (fires at MaxDepth), not cycle-based — a non-IId cycle in an otherwise shallow graph still requires reaching the depth limit to be caught. True cycle detection (visit-set tracking) for non-IId types is a separate enhancement; the originally-proposed [AcBinaryCircular] attribute or universal cycle-set tracking could close the remaining gap. Track as a future enhancement.

ACCORE-BIN-I-D9Y2: Default-value omission relies on type-level default consistency across writer/reader

Status: Open Affects: All property writes — BinaryTypeCode.PropertySkip (102) marker

The serializer writes a 1-byte PropertySkip marker for any property whose value equals default(T) instead of the full encoded value. The deserializer interprets this marker as "leave the target property at its CLR default" — which assumes the consumer-side type definition has the same default value as the producer-side.

Impact: Latent silent corruption across version-mismatched readers/writers. Three concrete failure modes:

1. Default-value change in type definition: bool IsActive = true refactored to bool IsActive = false → wire produced with the old default decodes to wrong value on the new reader.
2. Property added with non-zero default in v2: e.g., v2 adds int RetryCount = 3. Deserializing a v1 wire on the v2 reader → RetryCount = 0 (CLR default) instead of the expected v2 default 3.
3. Cross-language consumers: a non-.NET reader following the wire spec must implement the same default-resolution rules per type — not portable, not self-describing on the wire.

Wire-size impact (production-realistic): Significant for DTOs with many optional/default-valued fields (status flags, nullable refs, default ints/decimals/Guids/DateTimes). Benchmark contribution: estimated ~3-8% of AcBinary wire-size advantage vs MemoryPack on the current test data; real-world DTOs may see 10-30% depending on the default-value share.

Possible fix directions (decision deferred — see BINARY_TODO.md#accore-bin-t-w7n5):

• Doc-only: position as a deliberate protobuf-style feature, consumer responsibility to keep type definitions stable across versions.
• Option flag: AcBinarySerializerOptions.OmitDefaults (default true for back-compat); false writes every property's full value regardless. Lets consumers opt out for fragile-class-evolution scenarios.
• Hybrid: ship doc + flag, default true.

ACCORE-BIN-I-T7K3: MaxDepthBehavior.Truncate causes wire-misalignment on the SGen write path

Status: Open · Severity: Major (silent data corruption when opt-in to Truncate with SGen-enabled types) · Area: Source generator emit (AcBinarySourceGenerator.cs) + SGen-path marker writers (Serializers/Binaries/AcBinarySerializer.BinarySerializationContext.PropertyWriters.cs)

Description

With MaxDepthBehavior.Truncate opt-in AND UseGeneratedCode = true (SGen path), a cyclic graph that hits MaxDepth produces wire bytes that deserialize incorrectly — AcBinaryDeserializationException: [DECIMAL_DRIFT] or generic IndexOutOfRangeException raised by the SGen reader.

=========== ReferenceHandling: None, UseSgen: True, UseMeta: False ===========
AcBinarySerializer.Serialize(cyclicOrder, options-with-MaxDepth=10-and-Truncate);
// Serialize succeeds (truncation appears to write Null at the boundary)
order.BinaryTo<TestOrder_Circ_Ref>();
// Throws DECIMAL_DRIFT at pos=669

Runtime path (UseGeneratedCode = false) with the same Truncate setting works correctly. Throw and Disable on the SGen path are also unaffected — only the SGen + Truncate combination corrupts.

Root cause (suspected)

The check-before-marker placement that wires Truncate correctly was added to WriteObjectFullMarker* (in PropertyWriters.cs) and the SGen direct-marker emit patterns. Comparison with the runtime path (which works) suggests the SGen-side has either:

• A subtle inc/dec asymmetry between EnterRecursion / ExitRecursion on a path that doesn't go through WriteObjectFullMarker* (e.g. a poly-related branch),
• OR the wire-size difference between runtime (1705 B for the test case) and SGen (1449 B) for the same data indicates the SGen path encodes the cycle's truncation boundary in a slightly different position than the SGen reader expects.

Reproducer: AyCode.Core.Tests/Serialization/AcBinarySerializerIIdReferenceTests.cs → SameInstance_SerializeAndDeserialize (True, False) sub-test.

Known workaround

For Truncate use cases on SGen-typed graphs:

• Use UseGeneratedCode = false (runtime path) — works correctly. Trades SGen performance for Truncate correctness.
• Or keep MaxDepthBehavior = Throw (default) — fail-fast surfaces the cycle as an exception rather than truncating.

Related TODO

None yet — needs targeted debugging session to isolate the SGen-path wire diff.

Wire Format / Cross-platform

ACCORE-BIN-I-E4N9: Wire format is host-native-endian, NOT canonical little-endian

Status: Open Affects: Serializer + Deserializer — all primitive readers/writers (Int16/UInt16/Int32/UInt32/Int64/UInt64/Float32/Float64/Decimal/Guid etc.), H2Q6 string tier headers (StringSmall/Medium/Big charLen+utf8Len pack), WireMode.Fast UTF-16 raw memcopy, every Unsafe.WriteUnaligned<T> / Unsafe.ReadUnaligned<T> call site.

The serializer/deserializer write/read multi-byte fields via Unsafe.WriteUnaligned<T>(ref byte, T) / Unsafe.ReadUnaligned<T>(ref byte), which use host-native endianness. On little-endian hosts (x86, x64, ARM64, WASM) this happens to match the wire-format-canonical little-endian; on big-endian hosts (PowerPC big-endian, MIPS big-endian, IBM-Z / S390x, SPARC big-endian) the bytes are reversed. A wire produced on a big-endian host cannot be read by a little-endian reader (and vice versa).

Impact: Cross-platform serialization between hosts of different endianness is currently silently broken — payloads decode with byte-reversed integers, floats, and headers. Same-endianness round-trips work correctly. Affects H2Q6 string tier headers (charLen / utf8Len byte-swap), all multi-byte primitives, and WireMode.Fast UTF-16 raw payloads.

Currently supported NuGet release platforms — all little-endian:

• Cloud server (x64): Intel, AMD
• Desktop (x64): Intel, AMD
• Apple Silicon (ARM64): macOS, iOS, Mac Catalyst
• Blazor WASM (WASM SIMD spec mandates little-endian)

Currently NOT supported (would silently emit/read wrong-endian wire on these hosts):

• PowerPC big-endian (legacy IBM Power)
• MIPS big-endian (legacy embedded)
• IBM-Z / S390x (mainframe)
• SPARC big-endian (legacy Sun/Oracle)

Possible fix directions:

1. Document NuGet contract as little-endian-only — zero implementation cost, current code matches. NuGet readme + AcBinarySerializer XML doc-comment explicitly state the LE platform requirement; big-endian is "undefined behavior, wire silently incompatible". Pragmatic for the modern target platforms; matches the current de-facto reality.

2. Defensive endian-guard at every Unsafe.*Unaligned call site:

   if (BitConverter.IsLittleEndian)
       Unsafe.WriteUnaligned<ushort>(ref _buffer[pos], value);
   else
       BinaryPrimitives.WriteUInt16LittleEndian(_buffer.AsSpan(pos, 2), value);
   

   BitConverter.IsLittleEndian is [Intrinsic] and constant-folds at JIT/AOT — zero cost on LE hosts (the else branch is dead-code-eliminated). On big-endian hosts the BinaryPrimitives.*LittleEndian family does the byte-swap via BinaryPrimitives.ReverseEndianness. Adds conditional code at all primitive r/w sites + H2Q6 headers (~10-15 sites).

3. Replace all Unsafe.WriteUnaligned/ReadUnaligned with BinaryPrimitives.*LittleEndian unconditionally — these wrap Unsafe.WriteUnaligned + (conditional, JIT-folded on LE) ReverseEndianness. Slightly higher Span-creation overhead vs. direct ref byte writes; marginal perf cost on LE hosts. Over-engineering vs direction 2.

Pragmatic recommendation: Direction 1 (document) is the NuGet release minimum — explicitly state the LE constraint in the readme and XML doc-comments. Direction 2 (full endian-guard) is the comprehensive fix and should be a separate sprint covering ALL primitive r/w sites, H2Q6 string headers, WireMode.Fast UTF-16 path, and Float32/Float64/Decimal/Guid edge cases. A full BE-readiness audit is a non-trivial undertaking — only justified when there is concrete BE platform demand from a consumer.

Related TODO: worth tracking a follow-up TODO entry if direction 2 is chosen — covers writer + reader + SGen template + new round-trip tests on a BE-emulated host.

Cross-cutting (canonical home: ../XCUT/)

ACCORE-XCUT-I-X8Q1: JSON-in-Binary request parameters — cross-ref

Status: Closed (2026-04-26, see canonical entry).

Canonical entry: ../XCUT/XCUT_ISSUES.md#accore-xcut-i-x8q1. Summary: client→server request parameters previously used JSON inside a Binary envelope (SignalPostJsonDataMessage<T>); response path was already pure Binary. Migration landed in commits cdd54d3 + 3b70070 via SignalParams (length-prefixed binary pack/unpack) — wire is now Binary in both directions. Migration plan tracked in BINARY_TODO.md#accore-bin-t-s8p4 (also Closed).