Switch to BCL UTF-8 APIs for string (de)serialization
Replaced custom Utf8Transcoder logic with System.Text.Encoding.UTF8 and System.Text.Unicode.Utf8 for string encoding/decoding in AcBinarySerializer and AcBinaryDeserializer. PropertyMetadataBase now uses Encoding.UTF8.GetBytes for property name encoding. Retained Utf8Transcoder for any remaining SIMD/custom logic. No public API changes; internal refactoring for performance and maintainability.
Benchmark stabilization & charset-param workload support
Major overhaul of the custom benchmark harness:
- Per-serializer warmup, GC isolation, pilot discard, and CPU pinning for stable, reproducible results
- Adaptive per-cell iteration targeting (~250ms/sample) and statistical reporting (min/max/stddev/CV)
- CLI/menu support for single-cell A/B runs
- Test data refactored to ASCII baselines with configurable charset suffix (6 presets), selectable via menu; charset recorded in all outputs
- Markdown/console output now includes per-op µs, inter-sample range, CV warnings, and iteration counts
- Documentation updated with rationale, methodology, and notes on reverted/experimental optimizations
Enables reliable, cross-charset, release-grade performance measurement for AcBinary.
- Add overflow/corruption guards to string (de)serialization (writer/reader now throw on invalid lengths)
- Remove dead string serialization methods per BINARY_TODO.md audit
- Update BINARY_TODO.md with closure/resolution for H2Q6, O7G2, V4N5, and related entries
- Add MaxStringCharLength constant and update marker reservations in BinaryTypeCode
- Simplify string cache ASCII verification in deserializer
- Add SGen/Runtime round-trip compatibility tests for large/deep data
- Minor code modernization and style improvements
SIMD Utf8Transcoder.GetUtf8ByteCount + test suite
Introduced SIMD-accelerated Utf8Transcoder.GetUtf8ByteCount for efficient UTF-8 byte counting, replacing all writer-side Encoding.UTF8.GetByteCount usages. Added 29 unit tests for correctness across ASCII, Hungarian, CJK, emoji, and boundary cases. Updated benchmarks to ensure FixStr is bypassed and wire mode is selectable. Documented implementation and dead-code review in BINARY_TODO.md. No public API changes.
Fix Utf8Transcoder AVX2 bug, add SIMD boundary tests
- Added Hungarian language preference rule to copilot-instructions.md.
- Fixed AVX2 SIMD bug in Utf8Transcoder: corrected upper-half store offset from Vector128<ushort>.Count to Vector256<ushort>.Count, preventing memory overlap on 32+ byte ASCII runs.
- Added Utf8TranscoderTests covering all SIMD/scalar paths, with boundary and round-trip tests for ASCII, Hungarian, CJK, emoji, and mixed content, ensuring correctness and BCL compatibility.
Refactor: extract UTF-8 transcoder to Utf8Transcoder
Moved all UTF-8/UTF-16 encoding, decoding, and char counting logic from AcBinarySerializer/AcBinaryDeserializer into a new internal Utf8Transcoder class. Updated all call sites to use the new class. Removed redundant private methods from the original classes. Updated BINARY_TODO.md to clarify SIMD decode status and rationale for deferring AVX2 multi-byte SIMD path. No functional changes—pure refactor for maintainability and future SIMD work.
SIMD UTF-8 upgrades, i18n test data, MVC disabled
- Switch all test/benchmark data to Hungarian UTF-8 strings for i18n coverage
- Add AVX-512BW, Vector256, and Vector128 SIMD paths for UTF-8/UTF-16 encode/decode (ASCII and multi-byte) in binary serializer/deserializer
- Update WireMode docs for encoding guidance per workload/host
- Block-comment and disable MVC formatters and Microsoft.AspNetCore.App reference due to .NET 10 Hybrid client conflict; update docs to reflect temporary state
- Update appsettings: replace WaitForFlush with FlushPolicy
- Revise BINARY_TODO.md for SIMD transcoder progress and next steps
Add docs for AcBinary MVC formatters and pipeline updates
Comprehensive documentation for new ASP.NET Core MVC formatters supporting AcBinary, including registration, media type, request/response flow, error handling, and future plans. Updated project and topic docs to reference MVC formatters and folder structure. Added performance planning entry for StreamPipeWriter congestion fallback. Expanded markerless schema lane rationale and updated architecture docs to reflect MVC formatter integration. Improved navigation and layering documentation.
Remove PipeReader APIs from AcBinaryDeserializer
Refactored to remove all PipeReader-based async deserialization methods from AcBinaryDeserializer. Updated BINARY_TODO.md to clarify that draining PipeReader to AsyncPipeReaderInput is now a consumer responsibility. Refactored AcBinaryInputFormatter to inline the drain-loop and background deserialization, following new layering guidance. Updated comments and docs to reflect these changes.
AcBinary: Add ASCII string markers, doc optimizations
Enhanced string encoding with FixStrAscii/StringAscii markers for efficient ASCII handling, updated header flag base to 0xB0, and expanded documentation with marker-dispatch logic, performance results, and markerless schema lane plans.
AcBinary: ASCII string opt, Type-based API, MVC support
- Add ASCII-optimized string serialization/deserialization with new FixStrAscii/StringAscii markers for fast byte→char widening.
- Introduce non-generic Type-based Serialize/Deserialize overloads for runtime-typed scenarios (plugin, MVC, model binding).
- Add AcBinaryInputFormatter/OutputFormatter and AddAcBinaryFormatters extensions for ASP.NET Core MVC integration.
- Update project references and close ACCORE-BIN-T-N9G6 in docs.
Add issue/TODO for AcBinary default-value omission risk
Documented ACCORE-BIN-I-D9Y2 in BINARY_ISSUES.md, detailing the risk of silent data corruption when omitting default-valued properties if type defaults diverge between writer and reader. Added ACCORE-BIN-T-W7N5 in BINARY_TODO.md to track mitigation options, including a possible opt-out flag and required documentation/tests. Both entries are cross-referenced; decision on mitigation is deferred.
SIMD-accelerated UTF-8 encode/decode for AcBinary
- Added Vector256-based SIMD path for UTF-8 char counting in deserializer, replacing scalar loop for faster ASCII/multibyte handling.
- Introduced EncodeUtf8SinglePass in serializer: layered SIMD/DWORD/scalar UTF-16→UTF-8 encoding, bypassing Encoding.UTF8.GetBytes.
- Updated serializer to use new encoder for string writes.
- Expanded "fastestbyte" benchmark mode to compare both AcBinary (UTF-8/UTF-16) and MemoryPack strategies.
- Improved comments and docs to clarify new SIMD logic.
Disable ASCII fast paths; add FastestByte mode, plan tasks
Temporarily disable ASCII string fast paths in AcBinarySerializer and AcBinaryDeserializer to isolate and benchmark the custom UTF-8 encoder/decoder. Add "FastestByte" benchmark mode for focused AcBinary vs MemoryPack Byte[] comparison. Update BINARY_TODO.md with new technical tasks for .NET 11 SIMD decoder, sentinel-length encoding, ASCII marker-dispatch, and a custom UTF-8 encoder. These changes support staged optimization and future performance improvements.
Optimize string serialization: ASCII & UTF-8 fast paths
- Refactored AcBinarySerializer/Deserializer to use single-pass UTF-8 encoding and a custom allocation-free UTF-8 decoder, improving performance for both ASCII and non-ASCII strings.
- Expanded BinaryTypeCode with new ASCII string markers (FixStrAscii, StringAscii) and updated helpers for robust, branch-friendly string dispatch.
- Updated settings.local.json with new diagnostic and plugin management commands.
Optimize FastWire string (de)serialization and benchmarks
- Increased release benchmark iterations for more robust testing.
- Improved FastWire string deserialization with zero-copy UTF-16.
- Set FastWire and string caching options during context init/reset.
- Optimized FastWire string serialization for direct UTF-16 copy.
- Enhanced non-ASCII string fallback to use Utf8NoBom encoding.
- Refactored WriteFixStr for efficient ASCII and fallback handling.
Support FixStr for short non-ASCII strings (UTF-8)
Refactored WriteFixStrDirect to encode short non-ASCII strings as FixStr if their UTF-8 byte count is ≤31, not just ASCII. This improves efficiency for short international strings. Also ensured correct buffer sizing and direct UTF-8 encoding. No functional change to project file; property repositioned for clarity.
NativeAOT: full DAMs propagation, trimmer-safe serializers
- Propagate [DynamicallyAccessedMembers] from all public Serialize<T>/Deserialize<T> APIs through all type/property metadata and factories, centralizing requirements in TypeMetadataBase.RequiredMembers.
- Add [UnconditionalSuppressMessage] for known trimmer blind spots (polymorphism, inheritance, nested types) with detailed justifications.
- Update all internal delegate/factory signatures to preserve DAMs context.
- Annotate public APIs for AOT safety; document consumer requirements for SGen or rooted model assemblies.
- Update BINARY_FEATURES.md with NativeAOT/trimmer compatibility, guidance, and limitations.
- Adjust benchmark project for AOT/JIT parity and add i18n test data.
- No breaking API changes; SGen and Runtime paths remain, now fully AOT-compatible.
NativeAOT: fallback for delegates, exclude MessagePack
Added AYCODE_NATIVEAOT symbol for AOT builds and excluded MessagePack benchmarks from NativeAOT due to lack of AOT support. Updated AcSerializerCommon to use reflection-based delegates when dynamic code is unavailable, ensuring compatibility with both JIT and AOT. Added explanatory comments throughout.
Remove SegmentBufferReader; unify on AsyncPipeReaderInput
Migrates all SignalR chunked streaming receive logic to AsyncPipeReaderInput, fully removing SegmentBufferReader and SegmentBufferReaderInput from the codebase. Updates all references, deserialization paths, and documentation to reflect the new unified primitive. Marks ADR-0003 as accepted (partially executed), closes related TODOs, and clarifies protocol docs. Sets DoubleBuffered as the default FlushPolicy. No wire format or behavioral changes; all tests pass.
Switch to FlushPolicy enum for streaming flush control
Replaces the legacy bool waitForFlush with a new FlushPolicy enum (PerChunk, DoubleBuffered, Coalesced) across all binary streaming serialization APIs and SignalR protocol options. Updates all code, configuration, and documentation to use the new policy, clarifies memory/throughput trade-offs, and closes related TODOs. Stream-backed writers remain sequential; only parallel-capable Pipe-based writers honor the policy.
Move DrainFromAsync to tests, add in-memory benchmarks
- Moved AsyncPipeReaderInputExtensions.DrainFromAsync from the main framework to test-only assembly; no longer public API.
- Removed AcBinaryDeserializer.DeserializeFromPipeReaderAsync<T> from public API; tests now inline drain+deserialize logic.
- Added AcBinaryInMemoryPipeBenchmark and AcBinaryInMemoryRawByteArrayBenchmark to complete 2x2 transport × wire-format benchmark matrix.
- Refactored benchmark runner for interactive menu, settings, and CLI parsing.
- Expanded XML docs for AsyncPipeReaderInput and AsyncPipeWriterOutput to clarify push-pattern and real-world usage.
- Updated BINARY_ASYNCPIPE_TODO.md and related docs to reflect these changes.
Refactor pipe benchmarks to 2-task streaming model
Refactored AcBinaryNamedPipeBenchmark and AcBinaryNamedPipeRawByteArrayBenchmark to use a two-task (producer/consumer) streaming pipeline for deserialization, enabling true Ser↔Des overlap. Reduced BufferWriterChunkSize from 16K to 4K. Updated synchronization, cleanup, and comments to reflect the new architecture and improve performance comparison between chunked and raw byte[] modes.
Add raw NamedPipe benchmark & mux-mode AsyncPipe docs
- Add AcBinaryNamedPipeRawByteArrayBenchmark for raw NamedPipe (no chunking) to isolate kernel vs. AsyncPipe overhead
- Refactor progress reporting with in-place updates for all timed/allocation benchmarks
- Document [0xC8] marker as mux-mode direction; add ACCORE-BIN-T-M2X7 and ACCORE-BIN-I-C4N7 for multi-stream and single-consumer constraints
- Expand BINARY_WRITERS.md with parallel-flush regime analysis and allocation context
- Improve result comparison robustness for AsyncPipe-only runs
- Minor doc clarifications and explicit AsyncPipeReaderInput usage patterns
Refactor AcBinary SGen: switch dispatch + new TODOs
Refactored the AcBinary source generator to use switch-based dispatch for deserialization, replacing sequential if-else chains for improved performance and clarity. Updated comments to document the new logic. Added several new feature and refactor TODOs in BINARY_TODO.md, including Memory/Span overloads, async Stream serialization modes, non-generic Type-based APIs, attribute-driven polymorphism, and a thread-safety fix for serializer options. Each TODO includes rationale and acceptance criteria.
Modernize benchmarks, simplify attributes, doc cleanup
- Benchmark output now reports per-op µs and KB/op; added helpers for unit conversion and updated all output formats and headers.
- Split SetupAllocBytes into SetupSerializeAllocBytes and SetupDeserializeAllocBytes for finer allocation reporting.
- Simplified [AcBinarySerializable] usage in test models to single-argument form.
- Edited documentation for clarity, brevity, and consistency; improved navigation, updated technical details, and harmonized terminology across .md files.
Add AsyncPipe streaming mode, doc split, and test data tweaks
- Add AsyncPipe-only streaming mode to benchmark suite (CLI/menu)
- Aggregate and display AcBinarySerializableAttribute flags in options output
- Raise IId-ref and repeated-string share in all test data to ~20%
- Use explicit AcBinarySerializable(false, true, ...) on all test models
- Split streaming I/O issues/TODOs into BINARY_ASYNCPIPE_ISSUES.md and BINARY_ASYNCPIPE_TODO.md
- Update README and references for new streaming doc structure
- Minor code and doc cleanups for clarity and accuracy
Centralize pipe chunk size and fix buffer reset race
Centralized pipe chunk size config for all AcBinary pipe benchmarks, ensuring app-level and kernel buffer sizes stay in sync. Updated AsyncPipeReaderInput.MessageDone to atomically reset both _readPos and _writePos, preventing stale buffer reads. Improved comments and applied AggressiveOptimization to key methods. Adjusted AcquireChunk to ensure wire chunk fits exactly, avoiding kernel fragmentation. Updated related tests and documentation.
Refactor AcBinary NamedPipe to long-lived multi-message mode
Refactored the AcBinary NamedPipe benchmark to use a single long-lived AsyncPipeReaderInput in multi-message mode, with one background drain task and synchronous deserialization per message. Buffer recycling is now signaled by the consumer via a new MessageDone() method, called in the deserializer's finally block, preventing producer-consumer races. Added IsCompleted property to AsyncPipeReaderInput. Increased release-mode benchmark iteration counts. Updated documentation and comments to reflect the new architecture and rationale.
Refactor AcBinary streaming: multi-message protocol
- Renamed framing flags to multiMessage for clarity in AsyncPipeReaderInput/AsyncPipeWriterOutput.
- Multi-message mode ([202]=end-of-message) now auto-resets input for reuse; session end is explicit.
- Updated framing state machine, buffer cycling, and sentinel logic.
- Revised all serializer/deserializer entry points and tests for new protocol.
- Expanded docs and XML comments to detail wire format and protocol constraints.
- Updated benchmarks and tests for new streaming API and multi-message behavior.
- Documented protocol limits and added security issue/TODO for type-name deserialization in SignalR binary protocol.
Refactor benchmarks; clarify AcBinary doc warnings
Refactored serializer benchmark infra for richer, structured results and added fresh/reused buffer writer scenarios for AcBinary and MemoryPack. Disabled AcBinary SGen for all test models to ensure runtime/reflection-only benchmarks. Updated documentation to clarify and cross-link all silent corruption risks (hash collisions, MaxDepth, PropertyFilter), harmonized warnings, and referenced relevant issue IDs for traceability.
Refactor AcBinaryHubProtocol for thread safety
- Removed shared _currentHeaderContext; header context is now passed as a parameter through Parse* and ReadArguments/ReadSingleArgument methods, and stored per-binder for chunked messages.
- Updated AyCodeBinaryHubProtocol to use the new header context flow for type resolution and argument deserialization.
- Added concurrency tests to verify protocol instance safety under multi-threaded use and prevent state corruption or type resolution races.
- Improved documentation and comments to clarify the stateless, concurrency-safe design.
Benchmark: multi-sample median timing & EH inlining docs
Added BenchmarkSamples for multi-sample median timing in benchmarks, reducing variance and improving result stability. Updated output to show sample count. Refactored RunTimed to support multiple samples. Expanded documentation on JIT inlining barriers: clarified that EH regions (try/catch/finally/using) in hot-path and generated methods block inlining on .NET 9, and provided guidance for future generator features and stackalloc usage. Added audit requirements for EH and stackalloc in hot paths.
Refactor AsyncPipeWriterOutput buffer management
Refactored AsyncPipeWriterOutput to lazily allocate and reuse the fallback ArrayPool<byte> buffer across a serialize lifecycle, releasing it only once at the end. Replaced the _ownedBuffer boolean with _hasOwnedBuffer and a nullable _ownedBuffer field. Centralized buffer release logic, updated diagnostics, and improved chunk acquisition to minimize ArrayPool churn and clarify buffer ownership semantics.
Separate raw and framed streaming in AcBinarySerializer
Refactored AcBinarySerializer and AsyncPipeWriterOutput to support both raw (headerless) and multiplexed/framed ([201][UINT16][data]) streaming wire formats, controlled by a new flag and explicit APIs. Updated AsyncPipeReaderInput and AcBinaryDeserializer to match, with new constructor options and documentation. Expanded tests for both modes and added runtime type detection for flush strategy safety. Minor refactoring and doc improvements throughout.
Remove NamedPipe helpers; make binary serializer transport-agnostic
Removed NamedPipe-specific serializer/deserializer helpers from the framework. Refactored tests to manage NamedPipe lifecycle directly using generic PipeWriter/PipeReader APIs. Introduced a transport-agnostic async deserialization method for any PipeReader. Updated XML docs and method signatures to clarify usage and flush strategies. All NamedPipe logic is now test-only; the core binary serializer is fully transport-agnostic. Minor test cleanups included.
Refactor AsyncPipeWriterOutput for stream compatibility
- Reduce test chunk size to 256 bytes and update test names/comments
- Add sender-side diagnostic logging and unify with receiver logs
- Detect StreamPipeWriter at runtime and enforce sequential flush/acquire for streams
- Retain parallelism for pipe-based writers (Kestrel/SignalR)
- Add DEBUG-only diagnostics at key chunking points
- Minor code style cleanups and doc clarifications
- Add Bash command to fetch StreamPipeWriter.cs for reference
Add ADR-0003: AcBinary streaming receive unification
- Add cross-cutting ADR-0003 for AsyncPipeReaderInput and transport-agnostic streaming helpers (NamedPipe, FileStream)
- Update BINARY and SIGNALR_BINARY_PROTOCOL docs to reference ADR-0003
- Add migration TODOs for each ADR-0003 step with acceptance criteria
- Add project ADR-0001 (binary projection serialization) and index
- Clarify buffer sizing and ADR refs in SegmentBufferReader docs
- Migrate JWT key/token log issues from LOGGING to new AUTH topic per ADR
- Update ADR template and improve doc formatting throughout
Refactor SIGDS docs, archive DEC log, add pipe tests
- Updated all references to AcSignalRDataSource docs to new SIGNALR_DATASOURCE/README.md location; introduced SIGDS topic and paired issues/TODO files.
- Implemented new Decision Log archival policy: last-15-active entries remain, older entries moved to year-month archive (LLMP-DEC-65, 67); updated docs-archive skill for two-rule rotation.
- Added new SIGDS architectural TODO (ACCORE-SIGDS-T-D9F2) for relocating DataSource code.
- Updated doc tables, glossaries, and conventions for SIGDS.
- Added AcBinarySerializerPipeParallelTests.cs for parallel serialization/deserialization round-trip tests.
Expand META-TODO scope; add BINARY_TODO entries; doc updates
- Broadened ACCORE-META-T-F8R3 to cover SKILL.md/registry text drift, not just summary staleness
- Added concrete SKILL.md drift examples and clarified fix direction
- Added BINARY_TODOs for Id detection (convention/attribute) and serializer-native ignore attribute
- Updated SIGNALR_BINARY_PROTOCOL_TODO.md and ADR 0001 to clarify deferral of decorator base/handshake TODOs
- Minor topic code length and JSON-in-Binary tech debt clarifications
- Synced references and cross-links with latest protocol decisions
Add META topic, CONSUMERS.md, and refine ADR/Framework rules
- Introduced META topic with META_ISSUES.md and META_TODO.md for protocol meta-tooling tracking.
- Updated TOPIC_CODES.md to register META topic.
- Added CONSUMERS.md to document consumer repos per Framework-First exception.
- Extended Framework-First rule to all docs, documenting the CONSUMERS.md exception.
- Refined adr-author skill: explicit ADR keyword triggers, complexity-based ask-back, and clarified non-trigger cases.
- Improves meta-tooling governance, cross-layer doc boundaries, and ADR process.
Update ID format to use per-repo prefixes and random suffix
Migrated all issue, TODO, and decision IDs to a new 4-part format: <PREFIX>-<TOPIC>-<TYPE>-<RAND>. Added per-repo prefix declarations in copilot-instructions.md and documented conventions in REPO_PREFIXES.md. Updated all topic registries, logs, cross-references, and documentation to use the new format. Introduced MIGRATION_ID_MAPPING.md for old-to-new ID mapping. Enhanced skills and protocol audit logic to validate and enforce per-repo prefixes and topic codes at runtime. Clarified Framework-First doctrine and ensured all references are unambiguous.
Add Rule #6, remove host folder, update deployment docs
- Introduced Rule #6 to all primary copilot-instructions.md files, enforcing authority checks, conservative rule application, and skill invocation over ad-hoc solutions.
- Removed the obsolete Mango/FruitBank/ deployment-host folder and its copilot-instructions.md; updated REPOS.md to reflect the new repo count.
- Migrated deployment-context documentation to Mango/Source/FruitBank/.github/copilot-instructions.md, preserving the relationship between the server source and plugin after folder removal.
ADR skill, status vocab, and security log fixes
- Enhanced `adr-author` skill and ADR template based on first use: clarified opinion sharing, cross-referencing, index updates, and parallel session handling; rewrote ADR template pointer and added first repo-wide ADR.
- Updated skill loading: only `docs-discovery` and `docs-check` pre-loaded; others lazy-loaded; removed `version` from SKILL.md frontmatters.
- Simplified issue/TODO status to `Open`, `InProgress`, `Closed`; updated archiving logic and migrated all entries.
- Added forward-slash glob pattern rule to `docs-discovery/SKILL.md`.
- Closed LOG-I-9 and LOG-I-10: DEBUG-gated sensitive log lines, fixed typo, and cross-referenced new ADR.
- Various documentation and index improvements.
Simplify Status field; add docs-archive skill & archiving
- Reduced Status field values in issues/TODOs to Open, InProgress, Closed; updated all affected entries to new convention
- Introduced docs-archive skill for rotating Closed entries into year-bucketed archive files; process is user-invoked or LLM-suggested, never automatic
- Expanded docs-discovery and protocol documentation to clarify archive file handling and on-demand loading
- Updated session setup: only reactive skills pre-loaded, user-gated skills now lazy-loaded for token efficiency
- Clarified and documented Status update workflow, archive eligibility, and lifecycle
- Updated all relevant issue/TODO files to match new Status conventions and archival process
Loretta