Refactor pipe benchmarks to 2-task streaming model
Refactored AcBinaryNamedPipeBenchmark and AcBinaryNamedPipeRawByteArrayBenchmark to use a two-task (producer/consumer) streaming pipeline for deserialization, enabling true Ser↔Des overlap. Reduced BufferWriterChunkSize from 16K to 4K. Updated synchronization, cleanup, and comments to reflect the new architecture and improve performance comparison between chunked and raw byte[] modes.
Add raw NamedPipe benchmark & mux-mode AsyncPipe docs
- Add AcBinaryNamedPipeRawByteArrayBenchmark for raw NamedPipe (no chunking) to isolate kernel vs. AsyncPipe overhead
- Refactor progress reporting with in-place updates for all timed/allocation benchmarks
- Document [0xC8] marker as mux-mode direction; add ACCORE-BIN-T-M2X7 and ACCORE-BIN-I-C4N7 for multi-stream and single-consumer constraints
- Expand BINARY_WRITERS.md with parallel-flush regime analysis and allocation context
- Improve result comparison robustness for AsyncPipe-only runs
- Minor doc clarifications and explicit AsyncPipeReaderInput usage patterns
Refactor AcBinary SGen: switch dispatch + new TODOs
Refactored the AcBinary source generator to use switch-based dispatch for deserialization, replacing sequential if-else chains for improved performance and clarity. Updated comments to document the new logic. Added several new feature and refactor TODOs in BINARY_TODO.md, including Memory/Span overloads, async Stream serialization modes, non-generic Type-based APIs, attribute-driven polymorphism, and a thread-safety fix for serializer options. Each TODO includes rationale and acceptance criteria.
Modernize benchmarks, simplify attributes, doc cleanup
- Benchmark output now reports per-op µs and KB/op; added helpers for unit conversion and updated all output formats and headers.
- Split SetupAllocBytes into SetupSerializeAllocBytes and SetupDeserializeAllocBytes for finer allocation reporting.
- Simplified [AcBinarySerializable] usage in test models to single-argument form.
- Edited documentation for clarity, brevity, and consistency; improved navigation, updated technical details, and harmonized terminology across .md files.
Add AsyncPipe streaming mode, doc split, and test data tweaks
- Add AsyncPipe-only streaming mode to benchmark suite (CLI/menu)
- Aggregate and display AcBinarySerializableAttribute flags in options output
- Raise IId-ref and repeated-string share in all test data to ~20%
- Use explicit AcBinarySerializable(false, true, ...) on all test models
- Split streaming I/O issues/TODOs into BINARY_ASYNCPIPE_ISSUES.md and BINARY_ASYNCPIPE_TODO.md
- Update README and references for new streaming doc structure
- Minor code and doc cleanups for clarity and accuracy
Centralize pipe chunk size and fix buffer reset race
Centralized pipe chunk size config for all AcBinary pipe benchmarks, ensuring app-level and kernel buffer sizes stay in sync. Updated AsyncPipeReaderInput.MessageDone to atomically reset both _readPos and _writePos, preventing stale buffer reads. Improved comments and applied AggressiveOptimization to key methods. Adjusted AcquireChunk to ensure wire chunk fits exactly, avoiding kernel fragmentation. Updated related tests and documentation.
Refactor AcBinary NamedPipe to long-lived multi-message mode
Refactored the AcBinary NamedPipe benchmark to use a single long-lived AsyncPipeReaderInput in multi-message mode, with one background drain task and synchronous deserialization per message. Buffer recycling is now signaled by the consumer via a new MessageDone() method, called in the deserializer's finally block, preventing producer-consumer races. Added IsCompleted property to AsyncPipeReaderInput. Increased release-mode benchmark iteration counts. Updated documentation and comments to reflect the new architecture and rationale.
Refactor AcBinary streaming: multi-message protocol
- Renamed framing flags to multiMessage for clarity in AsyncPipeReaderInput/AsyncPipeWriterOutput.
- Multi-message mode ([202]=end-of-message) now auto-resets input for reuse; session end is explicit.
- Updated framing state machine, buffer cycling, and sentinel logic.
- Revised all serializer/deserializer entry points and tests for new protocol.
- Expanded docs and XML comments to detail wire format and protocol constraints.
- Updated benchmarks and tests for new streaming API and multi-message behavior.
- Documented protocol limits and added security issue/TODO for type-name deserialization in SignalR binary protocol.
Refactor benchmarks; clarify AcBinary doc warnings
Refactored serializer benchmark infra for richer, structured results and added fresh/reused buffer writer scenarios for AcBinary and MemoryPack. Disabled AcBinary SGen for all test models to ensure runtime/reflection-only benchmarks. Updated documentation to clarify and cross-link all silent corruption risks (hash collisions, MaxDepth, PropertyFilter), harmonized warnings, and referenced relevant issue IDs for traceability.
Refactor AcBinaryHubProtocol for thread safety
- Removed shared _currentHeaderContext; header context is now passed as a parameter through Parse* and ReadArguments/ReadSingleArgument methods, and stored per-binder for chunked messages.
- Updated AyCodeBinaryHubProtocol to use the new header context flow for type resolution and argument deserialization.
- Added concurrency tests to verify protocol instance safety under multi-threaded use and prevent state corruption or type resolution races.
- Improved documentation and comments to clarify the stateless, concurrency-safe design.
Benchmark: multi-sample median timing & EH inlining docs
Added BenchmarkSamples for multi-sample median timing in benchmarks, reducing variance and improving result stability. Updated output to show sample count. Refactored RunTimed to support multiple samples. Expanded documentation on JIT inlining barriers: clarified that EH regions (try/catch/finally/using) in hot-path and generated methods block inlining on .NET 9, and provided guidance for future generator features and stackalloc usage. Added audit requirements for EH and stackalloc in hot paths.
Refactor AsyncPipeWriterOutput buffer management
Refactored AsyncPipeWriterOutput to lazily allocate and reuse the fallback ArrayPool<byte> buffer across a serialize lifecycle, releasing it only once at the end. Replaced the _ownedBuffer boolean with _hasOwnedBuffer and a nullable _ownedBuffer field. Centralized buffer release logic, updated diagnostics, and improved chunk acquisition to minimize ArrayPool churn and clarify buffer ownership semantics.
Separate raw and framed streaming in AcBinarySerializer
Refactored AcBinarySerializer and AsyncPipeWriterOutput to support both raw (headerless) and multiplexed/framed ([201][UINT16][data]) streaming wire formats, controlled by a new flag and explicit APIs. Updated AsyncPipeReaderInput and AcBinaryDeserializer to match, with new constructor options and documentation. Expanded tests for both modes and added runtime type detection for flush strategy safety. Minor refactoring and doc improvements throughout.
Remove NamedPipe helpers; make binary serializer transport-agnostic
Removed NamedPipe-specific serializer/deserializer helpers from the framework. Refactored tests to manage NamedPipe lifecycle directly using generic PipeWriter/PipeReader APIs. Introduced a transport-agnostic async deserialization method for any PipeReader. Updated XML docs and method signatures to clarify usage and flush strategies. All NamedPipe logic is now test-only; the core binary serializer is fully transport-agnostic. Minor test cleanups included.
Refactor AsyncPipeWriterOutput for stream compatibility
- Reduce test chunk size to 256 bytes and update test names/comments
- Add sender-side diagnostic logging and unify with receiver logs
- Detect StreamPipeWriter at runtime and enforce sequential flush/acquire for streams
- Retain parallelism for pipe-based writers (Kestrel/SignalR)
- Add DEBUG-only diagnostics at key chunking points
- Minor code style cleanups and doc clarifications
- Add Bash command to fetch StreamPipeWriter.cs for reference
Add ADR-0003: AcBinary streaming receive unification
- Add cross-cutting ADR-0003 for AsyncPipeReaderInput and transport-agnostic streaming helpers (NamedPipe, FileStream)
- Update BINARY and SIGNALR_BINARY_PROTOCOL docs to reference ADR-0003
- Add migration TODOs for each ADR-0003 step with acceptance criteria
- Add project ADR-0001 (binary projection serialization) and index
- Clarify buffer sizing and ADR refs in SegmentBufferReader docs
- Migrate JWT key/token log issues from LOGGING to new AUTH topic per ADR
- Update ADR template and improve doc formatting throughout
Refactor SIGDS docs, archive DEC log, add pipe tests
- Updated all references to AcSignalRDataSource docs to new SIGNALR_DATASOURCE/README.md location; introduced SIGDS topic and paired issues/TODO files.
- Implemented new Decision Log archival policy: last-15-active entries remain, older entries moved to year-month archive (LLMP-DEC-65, 67); updated docs-archive skill for two-rule rotation.
- Added new SIGDS architectural TODO (ACCORE-SIGDS-T-D9F2) for relocating DataSource code.
- Updated doc tables, glossaries, and conventions for SIGDS.
- Added AcBinarySerializerPipeParallelTests.cs for parallel serialization/deserialization round-trip tests.
Expand META-TODO scope; add BINARY_TODO entries; doc updates
- Broadened ACCORE-META-T-F8R3 to cover SKILL.md/registry text drift, not just summary staleness
- Added concrete SKILL.md drift examples and clarified fix direction
- Added BINARY_TODOs for Id detection (convention/attribute) and serializer-native ignore attribute
- Updated SIGNALR_BINARY_PROTOCOL_TODO.md and ADR 0001 to clarify deferral of decorator base/handshake TODOs
- Minor topic code length and JSON-in-Binary tech debt clarifications
- Synced references and cross-links with latest protocol decisions
Add META topic, CONSUMERS.md, and refine ADR/Framework rules
- Introduced META topic with META_ISSUES.md and META_TODO.md for protocol meta-tooling tracking.
- Updated TOPIC_CODES.md to register META topic.
- Added CONSUMERS.md to document consumer repos per Framework-First exception.
- Extended Framework-First rule to all docs, documenting the CONSUMERS.md exception.
- Refined adr-author skill: explicit ADR keyword triggers, complexity-based ask-back, and clarified non-trigger cases.
- Improves meta-tooling governance, cross-layer doc boundaries, and ADR process.
Update ID format to use per-repo prefixes and random suffix
Migrated all issue, TODO, and decision IDs to a new 4-part format: <PREFIX>-<TOPIC>-<TYPE>-<RAND>. Added per-repo prefix declarations in copilot-instructions.md and documented conventions in REPO_PREFIXES.md. Updated all topic registries, logs, cross-references, and documentation to use the new format. Introduced MIGRATION_ID_MAPPING.md for old-to-new ID mapping. Enhanced skills and protocol audit logic to validate and enforce per-repo prefixes and topic codes at runtime. Clarified Framework-First doctrine and ensured all references are unambiguous.
Add Rule #6, remove host folder, update deployment docs
- Introduced Rule #6 to all primary copilot-instructions.md files, enforcing authority checks, conservative rule application, and skill invocation over ad-hoc solutions.
- Removed the obsolete Mango/FruitBank/ deployment-host folder and its copilot-instructions.md; updated REPOS.md to reflect the new repo count.
- Migrated deployment-context documentation to Mango/Source/FruitBank/.github/copilot-instructions.md, preserving the relationship between the server source and plugin after folder removal.
ADR skill, status vocab, and security log fixes
- Enhanced `adr-author` skill and ADR template based on first use: clarified opinion sharing, cross-referencing, index updates, and parallel session handling; rewrote ADR template pointer and added first repo-wide ADR.
- Updated skill loading: only `docs-discovery` and `docs-check` pre-loaded; others lazy-loaded; removed `version` from SKILL.md frontmatters.
- Simplified issue/TODO status to `Open`, `InProgress`, `Closed`; updated archiving logic and migrated all entries.
- Added forward-slash glob pattern rule to `docs-discovery/SKILL.md`.
- Closed LOG-I-9 and LOG-I-10: DEBUG-gated sensitive log lines, fixed typo, and cross-referenced new ADR.
- Various documentation and index improvements.
Simplify Status field; add docs-archive skill & archiving
- Reduced Status field values in issues/TODOs to Open, InProgress, Closed; updated all affected entries to new convention
- Introduced docs-archive skill for rotating Closed entries into year-bucketed archive files; process is user-invoked or LLM-suggested, never automatic
- Expanded docs-discovery and protocol documentation to clarify archive file handling and on-demand loading
- Updated session setup: only reactive skills pre-loaded, user-gated skills now lazy-loaded for token efficiency
- Clarified and documented Status update workflow, archive eligibility, and lifecycle
- Updated all relevant issue/TODO files to match new Status conventions and archival process
- Introduced the `adr-author` skill for structured ADR creation; updated session setup and shared skills to require pre-loading it.
- Added `SKILL.md` and `ADR_TEMPLATE.md` for ADR authoring workflow and documentation.
- Updated protocol decision log with entries for the new skill and its integration.
- Documented two critical JWT logging security issues in `LOGGING_ISSUES.md`.
- Minor: added a cleanup Bash command in `settings.local.json`.
Refactor docs: topic folders, TOON, XCUT, protocol sync
- Migrated all topic documentation into dedicated folders with canonical `README.md`, `ISSUES.md`, and `TODO.md` per topic (e.g., `LOGGING/`, `SIGNALR/`, `BINARY/`, `TOON/`).
- Added comprehensive TOON serializer documentation: design, format, options, attributes, inference, issues, and TODOs.
- Introduced `XCUT` folder for cross-cutting issues and TODOs, with canonical entries and topic cross-references.
- Updated all references and navigation to use new folder-based doc paths; fixed links and clarified doc structure.
- Enhanced AI agent protocol: enforce session skill preloading, `[LOADED_DOCS: ...]` short-name prefix, and mandatory `docs-check` skill for doc/code sync.
- Updated `.csproj` to include all `README.md` files for IDE visibility.
- Improved and clarified SignalR, grid, and project-level documentation.
- Minor code/test tweaks and doc content corrections for consistency.
Update protocol, docs-discovery skill, and doc layering
- Switched AI AGENT CORE PROTOCOL to new `[LOADED_DOCS: N files (+K this turn: ...)]` prefix format in all primary instruction files; updated examples and enforcement details.
- Added `docs-discovery` skill for proactive .md doc loading before code search; documented usage and integration as a shared agent skill.
- Introduced `## Protocol History` section and `LLM_PROTOCOL_DECISIONS.md` decision log for cross-repo protocol change tracking.
- Expanded protocol-audit skill and `REPOS.md` to support 8 instruction files (primary/inherit split), new invariants, and known issues.
- Added/updated structured TODO and issues docs for serialization, logging, and SignalR binary protocol.
- Improved cross-references, doc layering, and clarified documentation-first coding workflow.
- Various minor doc clarifications and formatting for protocol consistency.
Introduced framework-first design rules and updated documentation to clarify framework vs. consumer boundaries. Added AcLoggerOptions and DI-based logger factory extensions to AyCode.Core, enabling per-category logger instantiation from appsettings.json. Standardized SignalR connection setup with AddAcDefaults, replacing project-specific code. Enhanced protocol configuration for DI scope isolation. Refactored appsettings.json structure and added MSBuild targets for config propagation. Removed obsolete code and updated comments to match new patterns.
Refactor SignalR protocol registration; add DI options
- Added AcSignalRServerProtocolExtensions and AcSignalRProtocolExtensions for idiomatic AddAcBinaryProtocol registration (server/client), using a shared BuildProtocol factory for DI/IOptions/inline config.
- Introduced AcHubConnectionOptions and AcSignalRConnectionExtensions for configuration-driven client setup.
- Refactored AcSignalRClientBase to require a preconfigured IHubConnectionBuilder, moving all connection/protocol config out of the base class.
- Removed legacy protocol constructors; all protocol instantiation is now options-based.
- Enforced WASM + AsyncSegment guard in AcBinaryHubProtocolOptions.Validate.
- Updated SIGNALR_BINARY_PROTOCOL.md and GLOSSARY.md for new DI/config patterns.
- Minor: updated settings.local.json with new DLL/plugin inspection commands.
- Added AcBinaryHubProtocolOptions for unified protocol configuration (serializer, mode, buffer size, flush strategy, timeout, name, logger) with validation and DI support.
- Refactored AcBinaryHubProtocol and AyCodeBinaryHubProtocol to use options object; legacy constructors now delegate to options-based API.
- Added per-chunk flush timeout to AsyncPipeWriterOutput and AcBinarySerializer; throws TimeoutException on slow consumers.
- Improved XML docs and comments for pipeline/backpressure/timeout clarity.
- Updated SIGNALR_BINARY_PROTOCOL.md to document new options and AsyncSegment platform rules.
- Simplified logging with null-conditional operators
- Temporarily disabled WASM AsyncSegment guard for testing
- Unified argument deserialization via GetArgBytes for zero-copy and pooled buffer support
- Removed DeserializeFromSequence in favor of new approach
- Applied improvements to AyCodeBinaryHubProtocol
- Updated comments and performed minor code cleanup
Add WASM detection and fallback to AcBinaryHubProtocol
Added a static IsBrowser flag to AcBinaryHubProtocol, initialized at type-load using OperatingSystem.IsBrowser(). The constructor now throws PlatformNotSupportedException if AsyncSegment mode is used on WebAssembly. Receive path adapts: skips background Task on WASM and deserializes synchronously on CHUNK_END. Updated logging and documentation to reflect browser-specific behavior.
Enhance logging for SignalR binary protocol and flush logic
- Added detailed LogInformation calls in AcBinaryHubProtocol for serialization/deserialization start/end, including chunked and non-chunked paths.
- Switched some LogInformation to LogDebug for chunk state and parsing to reduce verbosity.
- Improved chunked message logging with total sent size, chunk count, and data bytes.
- Removed .Forget() on _lastFlush in AsyncPipeWriterOutput to prevent double-await, with updated comments.
- Increased default Microsoft log level to Information in AcSignalRClientBase.
Raise SignalR client log level to Warning
Changed the minimum log level for Microsoft.Extensions.Logging in the SignalR client configuration from Debug to Warning. This reduces log verbosity by only outputting messages with a severity of Warning or higher.
Refactor AyCodeBinaryHubProtocol header logic
Refactored the per-message header to use a DataFlags enum, encoding data argument properties in a single byte for more expressive client handling. Introduced HeaderContext to encapsulate header state. Updated WriteHeader and ReadHeader to use the new format, and refactored ReadSingleArgument to support fast-paths for byte[], ConsumerDeserialize, and header-supplied types. Removed obsolete _currentSignalParams logic and improved documentation throughout.
Refactor SignalR protocol type resolution logic
Removed SignalParams.SignalDataType and migrated type resolution to protocol headers using new WriteHeader/ReadHeader extensibility hooks. AyCodeBinaryHubProtocol now writes and reads the concrete data argument type in the message header, enabling correct deserialization of object-typed arguments. Updated AcBinaryHubProtocol to support header context and made relevant helpers protected. Cleaned up legacy SignalDataType logic and improved documentation.
Refactor: replace PipeReaderBinaryInput with SegmentBufferReader
- Remove PipeReaderBinaryInput and all related code.
- Add SegmentBufferReader and SegmentBufferReaderInput for efficient, thread-safe chunked streaming deserialization.
- Update AcBinaryDeserializer and AcBinaryHubProtocol to use the new buffer for async segment protocol, improving state management and background deserialization.
- Enhance chunked protocol handling: skip re-presented chunk start frames, track consumed chunk frame bytes, and improve logging.
- Update test infrastructure to support async segment protocol and add AsyncSegmentPipeTransportWriter for realistic testing.
- Update settings.local.json to reflect new build/test commands and remove obsolete files.
- Improves performance, reliability, and testability of chunked SignalR streaming.
Added SKILL.md defining the protocol-audit skill to verify AI AGENT CORE PROTOCOL consistency across five AyCode/Mango repos. Introduced REPOS.md listing the repos, their absolute paths, and expected own-dep-repos for audit reference. These additions enable structured, cross-repo protocol compliance checks without direct file modification.
Refactor synchronous PipeReader/PipeWriter usage to avoid Task allocations via fast-path helpers. Add detailed debug/trace logging for chunked message flows and deserialization. Track active reads to prevent protocol errors. Refactor FindStreamedArgSlot and introduce ResolveStreamedArgType for dynamic streamed arg type resolution, with AyCodeBinaryHubProtocol override. Minor code cleanups and improved logging context throughout. Improves performance, correctness, and debuggability.
Implement self-describing chunked protocol ([201][UINT16][data], [202] end) for AsyncSegment mode, enabling true zero-copy, pipeline-parallel serialization/deserialization of large arguments in SignalR.
- AsyncPipeWriterOutput now reserves a 3-byte header per chunk and supports two backpressure modes.
- AcBinaryHubProtocol routes streamable arguments through WriteMessageChunked, with chunk accumulation and background deserialization on the receiver.
- Logging now uses ILogger; documentation and wire format details updated.
- Consumer code updated to use new mode and diagnostics.
- Improves throughput, memory usage, and maintainability for large payloads.
Refactored binary protocol to support three explicit serialization/transport strategies via BinaryProtocolMode: Bytes (byte[]), Segment (zerocopy PipeWriter), and AsyncSegment (async PipeWriter with pipeline parallelism). Updated AcBinaryHubProtocol and AyCodeBinaryHubProtocol to select serialization/deserialization paths based on mode. Improved documentation and XML comments to describe each mode's behavior and performance. DI registration now explicitly selects AsyncSegment mode for AyCodeBinaryHubProtocol. Default remains Bytes mode. These changes clarify protocol mechanics and enable better performance tuning.
Implements segment-level streaming for SignalR binary protocol via new AsyncPipeWriterOutput and PipeReaderBinaryInput types, enabling chunked serialization/deserialization directly over PipeWriter/PipeReader. Adds BinaryProtocolMode enum to select between standard and streaming modes. Updates protocol classes and documentation. Lays groundwork for future async streaming support.
Refactored AcBinaryHubProtocol and AyCodeBinaryHubProtocol to remove the VarUInt length prefix for raw byte[] arguments. Now, the protocol writes a tag (0x44) followed directly by the raw bytes, with argLength implying the payload size. Updated read logic to match: on detecting the tag, the code skips it and returns the remaining bytes as the payload. Updated documentation to clarify the new fast-path, protocol roles, and AcBinary detection. Set BufferWriterChunkSize to 4096 for SignalR in the base protocol for better alignment with Kestrel. Marked the related issue as resolved.
- Move SignalParams-aware deserialization logic from AcBinaryHubProtocol to new AyCodeBinaryHubProtocol, enabling project-specific customization.
- Make key deserialization and helper methods in AcBinaryHubProtocol protected and virtual for easier extension.
- Improve byte[] handling to distinguish between AcBinary-serialized and raw data.
- Remove diagnostic serialization verification from the base protocol.
- Update DI registration to use AyCodeBinaryHubProtocol with configurable options.
- Adjust client code to support object-based response data and raw byte handling.
- Comment out SignalResponseDataMessage diagnostic logger in Program.cs.
Increased InitialBufferCapacity default to 16 KB in AcBinarySerializerOptions and updated docs. Removed ProtocolRoundTripDiagnosticTest and related diagnostic code from SignalRClientToHubTest.cs. Cleaned up debug output in AcBinaryHubProtocol.cs by removing Debug.WriteLine statements.
- Add VarUIntSize and unsafe VarUInt writers for efficient buffer sizing and writing without redundant checks.
- Update WriteStringUtf8 to preallocate for VarUInt and string body in one step, reducing reallocations and risk of overflow.
- Change ArrayBinaryOutput default initial capacity to 65535.
- Use BufferWriterChunkSize from options in AcBinarySerializer.
- Fix typo in AcBinarySerializerOptions.
- Set SignalR client log level to Warning by default.
Add a production-faithful test harness for SignalR binary protocol, introducing SlabTransportWriter to simulate Kestrel's slab allocator and always force the BufferWriterOutput owned-buffer path. Add large-payload round-trip tests (including non-ASCII cases) to catch position drift and data corruption bugs. Enhance protocol tests to validate multi-segment output and byte-for-byte correctness. All protocol round-trips now exercise the multi-segment, non-array-backed buffer path.
TestMultiSegmentProtocol now uses a custom MemoryPool to simulate Kestrel's slab allocator with 256-byte segments for both writing and reading. This replaces manual multi-segment sequence creation with a real Pipe backed by SlabSimulatingPool, ensuring more realistic segment boundaries and offsets. Old helpers were removed, and comments updated to clarify the improved simulation of production SignalR/Kestrel pipe behavior.
Loretta