Extract ISerializerBenchmark to its own file
Moved ISerializerBenchmark from Program.cs to a new ISerializerBenchmark.cs file under the AyCode.Core.Serializers.Console.Benchmarks namespace. Updated all benchmark classes in Program.cs to implement the interface from the new namespace and made them internal. Added the necessary using directive to Program.cs. Adjusted a PowerShell script in settings.local.json to ensure the new using is present. Removed the old interface definition from Program.cs.
Refactor: move benchmark loop logic to BenchmarkLoop.cs
Refactored all benchmark execution infrastructure from Program.cs into a new internal static class BenchmarkLoop. This includes timing, allocation measurement, progress reporting, GC helpers, MemoryPack setup validation, and test data filtering. Updated Program.cs and all serializer benchmarks to use the new class. Added serAllocPct reporting in Output.cs and a PowerShell script for automated refactoring. No functional changes to benchmark logic.
Refactor: split Program.cs into Menu, Output, DTO
Refactored the benchmark console app for modularity:
- Moved all menu logic to Menu.cs (main/settings menus)
- Moved all output/result formatting to Output.cs
- Extracted BenchmarkResult DTO to BenchmarkResult.cs
- Program.cs now only handles orchestration and the benchmark loop
- Moved GetCurrentCharsetName to Configuration.cs
- Removed obsolete Warmup methods from serializers
No functional changes; improves clarity and maintainability.
Refactor: centralize config/state in Configuration.cs
Moved all benchmark configuration, mutable state, and attribute-flag aggregation from Program.cs to a new Configuration.cs static class. Updated all references in Program.cs and related benchmark classes to use Configuration.<value>. Removed the "profiler" CLI mode and its code. Updated README.md to reflect these changes. This improves maintainability and keeps Program.cs focused on orchestration and UX, with no changes to benchmark logic.
Refactor output, allocation, and summary logic in Program
- Switched if/else and range checks to C# switch expressions for clarity.
- Improved console progress display with cleaner line updates.
- Added Thread.Sleep after JIT pre-warmup for stable benchmarking.
- Enhanced allocation measurement for serializer/deserializer setup.
- Made options and summary output conditional and more consistent.
- Standardized string outputs and comparison headers.
- Improved comments, XML docs, and code style for maintainability.
- No changes to core algorithms; all changes are quality-of-life and output improvements.
Phase-isolated Ser/Des warmup & GC in benchmarks
Refactored benchmark loop to perform separate warmup and measurement for serialization and deserialization phases, with forced GC.Collect at each phase boundary for heap and cache isolation. Added ForceGcCollect() and new WarmupSerialize/WarmupDeserialize interface methods (with defaults). Updated output, documentation, and per-phase iteration handling for improved accuracy and clarity. Added detailed comments explaining rationale and effects.
AcBinary: add framing doc, buffer growth fixes, doc updates
- Added `BINARY_WHYUSE.md` for architectural framing and value proposition
- Updated `BINARY_FEATURES.md` and `README.md` to reference and prioritize the new doc
- Documented AsyncPipeWriterOutput chunk-size limitation and workarounds in `BINARY_ASYNCPIPE_ISSUES.md`
- Refactored buffer growth logic in `AcBinarySerializer.BinarySerializationContext.cs` to validate capacity after grow and throw clear exceptions on under-provisioning; removed dead method
- Fixed chunk size alignment bug in `AsyncPipeWriterOutput.cs` to prevent buffer under-provisioning
- Added `AYCODE_NATIVEAOT` build config support in `Program.cs`
- Improved documentation clarity and error diagnostics for streaming/buffered serialization edge cases
Enable per-type property filter opt-out in AcBinary
Adds EnablePropertyFilterFeature to AcBinarySerializableAttribute, allowing types to opt out of property filter codegen and runtime checks. Updates source generator, metadata, and runtime logic to honor this flag. Removes UsePropertyFilter constant; emission is now attribute-driven. Also optimizes string serialization for non-ASCII cases and refactors deserializer byte reads for trusted single-segment fast paths. Backward compatible: property filter remains enabled by default.
Use ReadOnlySequence<byte> in benchmarks for deserialization
Updated all AcBinary and MemoryPack benchmark deserialization and round-trip verification methods to use ReadOnlySequence<byte> overloads instead of byte[] or ToArray(). This ensures benchmarks exercise the production-realistic deserialization path (e.g., for SignalR/Pipe consumers) and aligns buffer writer semantics across serializers. Added comments to clarify intent. No business logic was changed.
Benchmark stabilization & charset-param workload support
Major overhaul of the custom benchmark harness:
- Per-serializer warmup, GC isolation, pilot discard, and CPU pinning for stable, reproducible results
- Adaptive per-cell iteration targeting (~250ms/sample) and statistical reporting (min/max/stddev/CV)
- CLI/menu support for single-cell A/B runs
- Test data refactored to ASCII baselines with configurable charset suffix (6 presets), selectable via menu; charset recorded in all outputs
- Markdown/console output now includes per-op µs, inter-sample range, CV warnings, and iteration counts
- Documentation updated with rationale, methodology, and notes on reverted/experimental optimizations
Enables reliable, cross-charset, release-grade performance measurement for AcBinary.
SIMD Utf8Transcoder.GetUtf8ByteCount + test suite
Introduced SIMD-accelerated Utf8Transcoder.GetUtf8ByteCount for efficient UTF-8 byte counting, replacing all writer-side Encoding.UTF8.GetByteCount usages. Added 29 unit tests for correctness across ASCII, Hungarian, CJK, emoji, and boundary cases. Updated benchmarks to ensure FixStr is bypassed and wire mode is selectable. Documented implementation and dead-code review in BINARY_TODO.md. No public API changes.
SIMD-accelerated UTF-8 encode/decode for AcBinary
- Added Vector256-based SIMD path for UTF-8 char counting in deserializer, replacing scalar loop for faster ASCII/multibyte handling.
- Introduced EncodeUtf8SinglePass in serializer: layered SIMD/DWORD/scalar UTF-16→UTF-8 encoding, bypassing Encoding.UTF8.GetBytes.
- Updated serializer to use new encoder for string writes.
- Expanded "fastestbyte" benchmark mode to compare both AcBinary (UTF-8/UTF-16) and MemoryPack strategies.
- Improved comments and docs to clarify new SIMD logic.
Disable ASCII fast paths; add FastestByte mode, plan tasks
Temporarily disable ASCII string fast paths in AcBinarySerializer and AcBinaryDeserializer to isolate and benchmark the custom UTF-8 encoder/decoder. Add "FastestByte" benchmark mode for focused AcBinary vs MemoryPack Byte[] comparison. Update BINARY_TODO.md with new technical tasks for .NET 11 SIMD decoder, sentinel-length encoding, ASCII marker-dispatch, and a custom UTF-8 encoder. These changes support staged optimization and future performance improvements.
Optimize FastWire string (de)serialization and benchmarks
- Increased release benchmark iterations for more robust testing.
- Improved FastWire string deserialization with zero-copy UTF-16.
- Set FastWire and string caching options during context init/reset.
- Optimized FastWire string serialization for direct UTF-16 copy.
- Enhanced non-ASCII string fallback to use Utf8NoBom encoding.
- Refactored WriteFixStr for efficient ASCII and fallback handling.
NativeAOT: full DAMs propagation, trimmer-safe serializers
- Propagate [DynamicallyAccessedMembers] from all public Serialize<T>/Deserialize<T> APIs through all type/property metadata and factories, centralizing requirements in TypeMetadataBase.RequiredMembers.
- Add [UnconditionalSuppressMessage] for known trimmer blind spots (polymorphism, inheritance, nested types) with detailed justifications.
- Update all internal delegate/factory signatures to preserve DAMs context.
- Annotate public APIs for AOT safety; document consumer requirements for SGen or rooted model assemblies.
- Update BINARY_FEATURES.md with NativeAOT/trimmer compatibility, guidance, and limitations.
- Adjust benchmark project for AOT/JIT parity and add i18n test data.
- No breaking API changes; SGen and Runtime paths remain, now fully AOT-compatible.
NativeAOT: fallback for delegates, exclude MessagePack
Added AYCODE_NATIVEAOT symbol for AOT builds and excluded MessagePack benchmarks from NativeAOT due to lack of AOT support. Updated AcSerializerCommon to use reflection-based delegates when dynamic code is unavailable, ensuring compatibility with both JIT and AOT. Added explanatory comments throughout.
Remove SegmentBufferReader; unify on AsyncPipeReaderInput
Migrates all SignalR chunked streaming receive logic to AsyncPipeReaderInput, fully removing SegmentBufferReader and SegmentBufferReaderInput from the codebase. Updates all references, deserialization paths, and documentation to reflect the new unified primitive. Marks ADR-0003 as accepted (partially executed), closes related TODOs, and clarifies protocol docs. Sets DoubleBuffered as the default FlushPolicy. No wire format or behavioral changes; all tests pass.
Switch to FlushPolicy enum for streaming flush control
Replaces the legacy bool waitForFlush with a new FlushPolicy enum (PerChunk, DoubleBuffered, Coalesced) across all binary streaming serialization APIs and SignalR protocol options. Updates all code, configuration, and documentation to use the new policy, clarifies memory/throughput trade-offs, and closes related TODOs. Stream-backed writers remain sequential; only parallel-capable Pipe-based writers honor the policy.
Move DrainFromAsync to tests, add in-memory benchmarks
- Moved AsyncPipeReaderInputExtensions.DrainFromAsync from the main framework to test-only assembly; no longer public API.
- Removed AcBinaryDeserializer.DeserializeFromPipeReaderAsync<T> from public API; tests now inline drain+deserialize logic.
- Added AcBinaryInMemoryPipeBenchmark and AcBinaryInMemoryRawByteArrayBenchmark to complete 2x2 transport × wire-format benchmark matrix.
- Refactored benchmark runner for interactive menu, settings, and CLI parsing.
- Expanded XML docs for AsyncPipeReaderInput and AsyncPipeWriterOutput to clarify push-pattern and real-world usage.
- Updated BINARY_ASYNCPIPE_TODO.md and related docs to reflect these changes.
Refactor pipe benchmarks to 2-task streaming model
Refactored AcBinaryNamedPipeBenchmark and AcBinaryNamedPipeRawByteArrayBenchmark to use a two-task (producer/consumer) streaming pipeline for deserialization, enabling true Ser↔Des overlap. Reduced BufferWriterChunkSize from 16K to 4K. Updated synchronization, cleanup, and comments to reflect the new architecture and improve performance comparison between chunked and raw byte[] modes.
Add raw NamedPipe benchmark & mux-mode AsyncPipe docs
- Add AcBinaryNamedPipeRawByteArrayBenchmark for raw NamedPipe (no chunking) to isolate kernel vs. AsyncPipe overhead
- Refactor progress reporting with in-place updates for all timed/allocation benchmarks
- Document [0xC8] marker as mux-mode direction; add ACCORE-BIN-T-M2X7 and ACCORE-BIN-I-C4N7 for multi-stream and single-consumer constraints
- Expand BINARY_WRITERS.md with parallel-flush regime analysis and allocation context
- Improve result comparison robustness for AsyncPipe-only runs
- Minor doc clarifications and explicit AsyncPipeReaderInput usage patterns
Modernize benchmarks, simplify attributes, doc cleanup
- Benchmark output now reports per-op µs and KB/op; added helpers for unit conversion and updated all output formats and headers.
- Split SetupAllocBytes into SetupSerializeAllocBytes and SetupDeserializeAllocBytes for finer allocation reporting.
- Simplified [AcBinarySerializable] usage in test models to single-argument form.
- Edited documentation for clarity, brevity, and consistency; improved navigation, updated technical details, and harmonized terminology across .md files.
Add AsyncPipe streaming mode, doc split, and test data tweaks
- Add AsyncPipe-only streaming mode to benchmark suite (CLI/menu)
- Aggregate and display AcBinarySerializableAttribute flags in options output
- Raise IId-ref and repeated-string share in all test data to ~20%
- Use explicit AcBinarySerializable(false, true, ...) on all test models
- Split streaming I/O issues/TODOs into BINARY_ASYNCPIPE_ISSUES.md and BINARY_ASYNCPIPE_TODO.md
- Update README and references for new streaming doc structure
- Minor code and doc cleanups for clarity and accuracy
Centralize pipe chunk size and fix buffer reset race
Centralized pipe chunk size config for all AcBinary pipe benchmarks, ensuring app-level and kernel buffer sizes stay in sync. Updated AsyncPipeReaderInput.MessageDone to atomically reset both _readPos and _writePos, preventing stale buffer reads. Improved comments and applied AggressiveOptimization to key methods. Adjusted AcquireChunk to ensure wire chunk fits exactly, avoiding kernel fragmentation. Updated related tests and documentation.
Refactor AcBinary NamedPipe to long-lived multi-message mode
Refactored the AcBinary NamedPipe benchmark to use a single long-lived AsyncPipeReaderInput in multi-message mode, with one background drain task and synchronous deserialization per message. Buffer recycling is now signaled by the consumer via a new MessageDone() method, called in the deserializer's finally block, preventing producer-consumer races. Added IsCompleted property to AsyncPipeReaderInput. Increased release-mode benchmark iteration counts. Updated documentation and comments to reflect the new architecture and rationale.
Refactor AcBinary streaming: multi-message protocol
- Renamed framing flags to multiMessage for clarity in AsyncPipeReaderInput/AsyncPipeWriterOutput.
- Multi-message mode ([202]=end-of-message) now auto-resets input for reuse; session end is explicit.
- Updated framing state machine, buffer cycling, and sentinel logic.
- Revised all serializer/deserializer entry points and tests for new protocol.
- Expanded docs and XML comments to detail wire format and protocol constraints.
- Updated benchmarks and tests for new streaming API and multi-message behavior.
- Documented protocol limits and added security issue/TODO for type-name deserialization in SignalR binary protocol.
Refactor benchmarks; clarify AcBinary doc warnings
Refactored serializer benchmark infra for richer, structured results and added fresh/reused buffer writer scenarios for AcBinary and MemoryPack. Disabled AcBinary SGen for all test models to ensure runtime/reflection-only benchmarks. Updated documentation to clarify and cross-link all silent corruption risks (hash collisions, MaxDepth, PropertyFilter), harmonized warnings, and referenced relevant issue IDs for traceability.
Refactor AcBinaryHubProtocol for thread safety
- Removed shared _currentHeaderContext; header context is now passed as a parameter through Parse* and ReadArguments/ReadSingleArgument methods, and stored per-binder for chunked messages.
- Updated AyCodeBinaryHubProtocol to use the new header context flow for type resolution and argument deserialization.
- Added concurrency tests to verify protocol instance safety under multi-threaded use and prevent state corruption or type resolution races.
- Improved documentation and comments to clarify the stateless, concurrency-safe design.
Benchmark: multi-sample median timing & EH inlining docs
Added BenchmarkSamples for multi-sample median timing in benchmarks, reducing variance and improving result stability. Updated output to show sample count. Refactored RunTimed to support multiple samples. Expanded documentation on JIT inlining barriers: clarified that EH regions (try/catch/finally/using) in hot-path and generated methods block inlining on .NET 9, and provided guidance for future generator features and stackalloc usage. Added audit requirements for EH and stackalloc in hot paths.
- Add TypeName to TestDataSet for clearer test scenario reporting
- Display serializer options in console and log outputs
- Extend BenchmarkResult with OptionsDescription
- Serializer benchmarks now provide detailed config summaries
- Log files now include test type and serializer options summary
- Generate .LLM Markdown results for LLM consumption and docs
- Reference .LLM results in BINARY_IMPLEMENTATION.md for visibility
Updated test and benchmark project dependencies to latest versions (BenchmarkDotNet, coverlet.collector, MSTest, etc.). Reformatted source generator project references for consistency. Commented out most AcBinary benchmark variants in Program.cs, leaving only FastMode+Default enabled. Significantly expanded README with detailed architecture, file descriptions, and configuration for each serializer subfolder. Added LLM maintenance notes to ensure documentation stays in sync with code changes.
Split WriteObjectProperties into markerless and metadata variants for clarity and performance. Adjust method inlining attributes to favor hot path optimization. Comment out WritePropertyValue and some AcBinaryBenchmark variants to streamline code and benchmarks. Improves maintainability and serialization efficiency.
Refactored BinaryTypeCode to reserve 0..63 for FixObj slot indices, enabling direct array access for object wrappers. Introduced a new polymorphic type prefix system for properties whose runtime type differs from their declared type, with first/repeated occurrence markers and combined ref-tracking support. Unified wrapper slot caching for SGen and runtime types, improving performance and eliminating dictionary lookups in hot paths. Updated code generation, tests, and constants to use the new slot system. Added new settings and utility scripts. Overall, serialization is now faster, more robust, and extensible.
Refactor serialization context to use precomputed boolean flags
(HasRefHandling, HasAllRefHandling, HasStringInterning) for faster
reference and string interning checks, replacing repeated enum
comparisons. Update source generator to emit code using these flags.
Add AcBinarySerializer.ScanOnly for isolated scan benchmarking.
Set MaxDepth in test options. Improves performance and maintainability.
Introduce new test models for circular refs, update tests to stress reference handling, and enhance deserializer to support ObjectRefFirst/WithMetadataRefFirst type codes. Fix intern cache index assignment, track generated readers in TypeMetadataWrapper, and disable UseGeneratedCode by default. Update benchmarks for reliability and diagnostics. These changes strengthen reference resolution, circular ref support, and performance.
- Refined collection kind mapping: IList<T>/IReadOnlyList<T> now treated as "IndexedCollection" for codegen, distinct from List<T>
- Generated code uses CollectionsMarshal.AsSpan for List<T> iteration, improving performance
- Updated generated read/write logic for collections to match new distinctions
- Added System.Runtime.InteropServices to generated code for span support
- Increased test iteration count in Program.cs for more robust benchmarks
- Enabled source-generated binary serialization for all test models by setting [AcBinarySerializable(true)]
- Set `[AcBinarySerializable(true)]` on all SharedTestModels types
- Default `UseMetadata` to true for property hash footer
- Reset slotted ID maps and cache indices in serializer context
- Reduce test iterations for DEBUG builds to speed up runs
- Add debug comments in context `Clear()` method
Added compile-time scan requirement flags to SerializableClassInfo and PropInfo, and implemented recursive analysis to determine if scan work is needed for reference tracking and string interning. Updated code generation to emit scan code only when necessary, with runtime guards based on compile-time analysis. Changed AcBinarySerializerOptions.UseMetadata default to false. Increased JIT wait in Program.cs for more reliable benchmarking. These changes reduce unnecessary scan calls and improve performance.
- Generate writers for nested types using flat class names (Outer_Inner_Leaf) to ensure uniqueness and validity.
- Apply property filters in generated code for all non-markerless properties, matching runtime behavior.
- Emit skip labels for each property in generated code for correct control flow.
- Remove PropertyFilter check from IsDirectObjectWrite; generated code now handles filtering.
- Change default ReferenceHandlingMode to All.
- Make BinaryPropertyFilterContext constructor public.
- Increase release warmup iterations in Program.cs from 3000 to 5000.
- Always write length prefix before ASCII and UTF-8 strings in AcBinarySerializer for consistency and correctness.
- Increase release warmup iterations from 2000 to 3000 in Program.cs.
Refactor AcBinarySerializer to use ReadOnlySpan<T> for bulk writing of primitive arrays and List<T>, replacing multiple specialized methods with a single TryWritePrimitiveCollection. This improves efficiency and reduces code duplication. Change default string interning mode to Attribute (opt-in). Update generated code path to allow reference tracking but not string interning. Adjust benchmarks to test correct serializer options. Reorder options for clarity.
- Integrated MemoryPack as a serializer in the benchmark suite
- Added [MemoryPackable] and [MemoryPackIgnore] to test models
- Enabled AcBinary source generation by default
- Updated benchmark app to include MemoryPack and focus on key serializers
- Added MemoryPack NuGet references to projects
- Refactored AcBinarySerializer.WriteString flag handling
Implement write plan mechanism for string interning and IId object reference tracking. Scan pass now builds pre-computed WriteDuplicateEntry instructions, eliminating hot path IdentityMap lookups and redundant getter calls in the write pass. Update BinarySerializationContext, tracking visit indices and managing write plan array. Refactor ScanInternString and TryTrack methods to record visit indices and build write instructions for all duplicate occurrences. Update write pass logic to consume write plan entries. Add debug validation for scan/write pass order. Update benchmarks and test harness. Set UseGeneratedCode default to false. Improves performance for scenarios with interning and reference tracking.
Introduce JitDisassemblyBenchmark for analyzing JIT-generated x64 assembly of AcBinarySerializer hot paths, accessible via --jitasm. Refactor string interning logic to support per-property and string collection interning, adding IsStringCollectionProperty and ScanStringCollection. Update ScanPass and WriteString for finer-grained control. Remove DEBUG-only CurrentPropertyPath in favor of a more robust property tracking approach. Update usage instructions and clean up related code.
- Respect both global and property-level [AcStringIntern] settings for string interning
- Add UseStringPropertyInterning method and flag-based caching in property accessors for fast runtime checks
- Update scan and write passes to use property-level interning decisions
- Introduce FilteredReferenceProperties for efficient scan filtering in TypeMetadataWrapper
- Refactor benchmarks to use correct serializer options
- Add TODOs and minor cleanups for clarity and future improvements
Pre-caches TypeMetadataWrapper instances for complex properties,
eliminating repeated GetWrapper dictionary lookups in serialization
and deserialization. Adds ComplexPropertyIndex and ComplexPropertyCount
fields, and PropertyTypeWrappers array to TypeMetadataWrapper. Refactors
scan, write, and populate passes to use cached wrappers, improving
performance for deep and polymorphic object graphs. Updates benchmarks
to focus on FastMode variants. No breaking changes; internal efficiency
improved.
- Refactor collection scan to pre-cache element wrappers and optimize ScanItem for polymorphic types
- Add DisableStringCaching to deserializer; call it on first interned string marker
- Update benchmarks to restore default and no-ref variants, clarify string interning options
- Ensure property scanning respects property filters, skipping filtered properties
Removed Newtonsoft.Json from benchmarks and codebase. Added AcBinaryBufferWriterBenchmark using ArrayBufferWriter and AcBinarySerializer's buffer writer API. Optimized WriteStringUtf8 for ASCII fast path. Improved ArrayBinaryOutput buffer reuse and memory management. Introduced Reset method to IBinaryOutputBase and implemented it in outputs. Streamlined serializer benchmarks to focus on AcBinary and System.Text.Json.
Loretta