Refactor AyCodeBinaryHubProtocol header logic
Refactored the per-message header to use a DataFlags enum, encoding data argument properties in a single byte for more expressive client handling. Introduced HeaderContext to encapsulate header state. Updated WriteHeader and ReadHeader to use the new format, and refactored ReadSingleArgument to support fast-paths for byte[], ConsumerDeserialize, and header-supplied types. Removed obsolete _currentSignalParams logic and improved documentation throughout.
Refactor SignalR protocol type resolution logic
Removed SignalParams.SignalDataType and migrated type resolution to protocol headers using new WriteHeader/ReadHeader extensibility hooks. AyCodeBinaryHubProtocol now writes and reads the concrete data argument type in the message header, enabling correct deserialization of object-typed arguments. Updated AcBinaryHubProtocol to support header context and made relevant helpers protected. Cleaned up legacy SignalDataType logic and improved documentation.
Refactor: replace PipeReaderBinaryInput with SegmentBufferReader
- Remove PipeReaderBinaryInput and all related code.
- Add SegmentBufferReader and SegmentBufferReaderInput for efficient, thread-safe chunked streaming deserialization.
- Update AcBinaryDeserializer and AcBinaryHubProtocol to use the new buffer for async segment protocol, improving state management and background deserialization.
- Enhance chunked protocol handling: skip re-presented chunk start frames, track consumed chunk frame bytes, and improve logging.
- Update test infrastructure to support async segment protocol and add AsyncSegmentPipeTransportWriter for realistic testing.
- Update settings.local.json to reflect new build/test commands and remove obsolete files.
- Improves performance, reliability, and testability of chunked SignalR streaming.
Added SKILL.md defining the protocol-audit skill to verify AI AGENT CORE PROTOCOL consistency across five AyCode/Mango repos. Introduced REPOS.md listing the repos, their absolute paths, and expected own-dep-repos for audit reference. These additions enable structured, cross-repo protocol compliance checks without direct file modification.
Refactor synchronous PipeReader/PipeWriter usage to avoid Task allocations via fast-path helpers. Add detailed debug/trace logging for chunked message flows and deserialization. Track active reads to prevent protocol errors. Refactor FindStreamedArgSlot and introduce ResolveStreamedArgType for dynamic streamed arg type resolution, with AyCodeBinaryHubProtocol override. Minor code cleanups and improved logging context throughout. Improves performance, correctness, and debuggability.
Implement self-describing chunked protocol ([201][UINT16][data], [202] end) for AsyncSegment mode, enabling true zero-copy, pipeline-parallel serialization/deserialization of large arguments in SignalR.
- AsyncPipeWriterOutput now reserves a 3-byte header per chunk and supports two backpressure modes.
- AcBinaryHubProtocol routes streamable arguments through WriteMessageChunked, with chunk accumulation and background deserialization on the receiver.
- Logging now uses ILogger; documentation and wire format details updated.
- Consumer code updated to use new mode and diagnostics.
- Improves throughput, memory usage, and maintainability for large payloads.
Refactored binary protocol to support three explicit serialization/transport strategies via BinaryProtocolMode: Bytes (byte[]), Segment (zerocopy PipeWriter), and AsyncSegment (async PipeWriter with pipeline parallelism). Updated AcBinaryHubProtocol and AyCodeBinaryHubProtocol to select serialization/deserialization paths based on mode. Improved documentation and XML comments to describe each mode's behavior and performance. DI registration now explicitly selects AsyncSegment mode for AyCodeBinaryHubProtocol. Default remains Bytes mode. These changes clarify protocol mechanics and enable better performance tuning.
Implements segment-level streaming for SignalR binary protocol via new AsyncPipeWriterOutput and PipeReaderBinaryInput types, enabling chunked serialization/deserialization directly over PipeWriter/PipeReader. Adds BinaryProtocolMode enum to select between standard and streaming modes. Updates protocol classes and documentation. Lays groundwork for future async streaming support.
Refactored AcBinaryHubProtocol and AyCodeBinaryHubProtocol to remove the VarUInt length prefix for raw byte[] arguments. Now, the protocol writes a tag (0x44) followed directly by the raw bytes, with argLength implying the payload size. Updated read logic to match: on detecting the tag, the code skips it and returns the remaining bytes as the payload. Updated documentation to clarify the new fast-path, protocol roles, and AcBinary detection. Set BufferWriterChunkSize to 4096 for SignalR in the base protocol for better alignment with Kestrel. Marked the related issue as resolved.
- Move SignalParams-aware deserialization logic from AcBinaryHubProtocol to new AyCodeBinaryHubProtocol, enabling project-specific customization.
- Make key deserialization and helper methods in AcBinaryHubProtocol protected and virtual for easier extension.
- Improve byte[] handling to distinguish between AcBinary-serialized and raw data.
- Remove diagnostic serialization verification from the base protocol.
- Update DI registration to use AyCodeBinaryHubProtocol with configurable options.
- Adjust client code to support object-based response data and raw byte handling.
- Comment out SignalResponseDataMessage diagnostic logger in Program.cs.
Increased InitialBufferCapacity default to 16 KB in AcBinarySerializerOptions and updated docs. Removed ProtocolRoundTripDiagnosticTest and related diagnostic code from SignalRClientToHubTest.cs. Cleaned up debug output in AcBinaryHubProtocol.cs by removing Debug.WriteLine statements.
- Add VarUIntSize and unsafe VarUInt writers for efficient buffer sizing and writing without redundant checks.
- Update WriteStringUtf8 to preallocate for VarUInt and string body in one step, reducing reallocations and risk of overflow.
- Change ArrayBinaryOutput default initial capacity to 65535.
- Use BufferWriterChunkSize from options in AcBinarySerializer.
- Fix typo in AcBinarySerializerOptions.
- Set SignalR client log level to Warning by default.
Add a production-faithful test harness for SignalR binary protocol, introducing SlabTransportWriter to simulate Kestrel's slab allocator and always force the BufferWriterOutput owned-buffer path. Add large-payload round-trip tests (including non-ASCII cases) to catch position drift and data corruption bugs. Enhance protocol tests to validate multi-segment output and byte-for-byte correctness. All protocol round-trips now exercise the multi-segment, non-array-backed buffer path.
TestMultiSegmentProtocol now uses a custom MemoryPool to simulate Kestrel's slab allocator with 256-byte segments for both writing and reading. This replaces manual multi-segment sequence creation with a real Pipe backed by SlabSimulatingPool, ensuring more realistic segment boundaries and offsets. Old helpers were removed, and comments updated to clarify the improved simulation of production SignalR/Kestrel pipe behavior.
Enhances debugging of custom binary serialization/deserialization in SignalR by introducing DiagnosticLogger hooks in both AcBinaryDeserializer and AcBinaryHubProtocol. Adds DEBUG-only verification methods to compare array-based and multi-segment deserialization, as well as IBufferWriter and byte[] serialization outputs, logging mismatches for easier bug isolation. Diagnostic loggers are automatically integrated with the hub and client loggers. Also includes extra debug output and a commented workaround for a known serialization issue. Diagnostics are opt-in and only active in DEBUG builds.
Moved BenchmarkTestDataProvider and TestDataSet to AyCode.Core.Tests.TestModels with public accessibility. Refactored dataset creation methods to accept a resetId parameter, allowing control over TestDataFactory ID resets. Improved code structure, formatting, and documentation for maintainability. The provider is now more flexible and easier to use in tests.
Introduce diagnostic and test infrastructure for SignalR binary protocol serialization/deserialization, including:
- ProtocolRoundTripDiagnosticTest for isolated protocol byte inspection
- TestMultiSegmentProtocol to exercise multi-segment buffer parsing
- TestInvocationBinder for correct parameter type binding
- Updates to TestableSignalRClient2 and TestableSignalRHub2 to route all messages through protocol round-trip
- Enhanced SendMessageToClient to simulate real SignalR transport
- Clarified SequenceBinaryInput segment handling logic
- Made TryParseMessage virtual in AcBinaryHubProtocol for testability
These changes improve test coverage for cross-boundary and multi-segment scenarios in SignalR message handling.
- Rewrote SequenceBinaryInput for lazy TryGet iteration (no segment array allocation), zero-copy access to segment backing arrays, and efficient cross-boundary reads using a reusable ArrayPool scratch buffer.
- Added Release() to IBinaryInputBase; now always called after deserialization to return scratch buffer.
- BufferWriterChunkSize is now mutable; set to 4096 for SignalR protocol for better pipe alignment.
- Added and updated documentation: detailed input buffer lifecycle, cross-boundary handling, and new BINARY_ISSUES.md and SIGNALR_ISSUES.md for known limitations and planned optimizations.
- No breaking API changes; improves performance, memory usage, and diagnostics for multi-segment binary deserialization.
- Change OnReceiveMessage signature to use `object data` (was `SignalData`), enabling type-aware and raw byte[] payloads.
- Implement three-path argument deserialization: byte[] fast-path, IsRawBytesData, and eager typed deserialization via SignalDataType.
- Add SignalDataType and IsRawBytesData fields to SignalParams for protocol guidance.
- Write path now uses AcBinarySerializer zero-copy to pipe; byte[] uses fast-path.
- SequenceBinaryInput now dynamically sizes scratch buffer for large cross-segment reads.
- Deserializer now advances segments before throwing end-of-buffer, improving multi-segment support.
- Set client logging to Debug for better diagnostics.
- Update all docs and markdown to reflect new protocol, dispatch model, and field semantics.
- AyCodeBinaryHubProtocol is now an empty derived class for registration/future hooks; SignalData is no longer the primary payload type.
- SignalResponseDataMessage is now an internal DTO with RawResponseData as object? (typed or byte[]), and GetResponseData<T>() is a direct cast.
Implements a high-performance raw byte[] protocol path for SignalR DataSource GetAll/LoadDataSource, using a new IsRawBytesData flag in SignalParams. When enabled, the server pre-serializes response data and sends it as a byte array, which the protocol passes through without further (de)serialization. The client receives the raw bytes and deserializes as needed, avoiding double serialization/deserialization and improving performance for large payloads.
Adds SerializerType selection to DataSource, propagates SignalParams through hub and protocol layers, and updates client/server/test code to support the new path. Also includes diagnostics flags for binary serialization debugging and fixes for multi-segment buffer handling.
Major overhaul for SignalR response pipeline:
- All deserialization now uses byte[] (offset/length) for zero-copy, allocation-free operation; all span/memory overloads removed.
- SignalR protocol sends (signalParams, object) directly; SignalData envelope and related logic removed.
- Server sets SignalParams.SignalDataType so protocol deserializes to the correct runtime type on the client.
- SignalResponseDataMessage now only used for client request/response tracking and stream path; RawResponseData holds the actual object.
- All extension methods, helpers, and infrastructure updated to use new byte[]-based APIs.
- AcSignalRDataSource and all test/benchmark code updated for new object flow.
- Removes all diagnostics, logging, and error handling related to binary envelopes.
- Enables true zero-copy, type-safe, allocation-free SignalR response handling.
Major protocol refactor: all byte[] payloads in SignalR hub/client interfaces, plumbing, and DTOs are now wrapped in SignalData, a disposable, ArrayPool-backed type with Span access. Introduces AyCodeBinaryHubProtocol (derived from AcBinaryHubProtocol) to rent pooled buffers for SignalData on receive. All message signatures, diagnostics, and serialization logic updated. Documentation and tests revised to reflect SignalData usage. Enables zero-copy, low-GC, high-performance binary messaging for large payloads.
Major protocol/API change: replace SignalReceiveParams with SignalParams everywhere. SignalParams now carries packed method parameters as a single byte[] and provides SetParameterValues/GetParameterValues for type-safe packing/unpacking. All hub/client interfaces, method signatures, and dispatch logic updated. Legacy parameter serialization helpers removed; all parameter logic is encapsulated in SignalParams. Documentation and tests updated to reflect new wire format and flow. This unifies parameter handling, clarifies the protocol, and enables robust, extensible type-guided serialization. Breaking change.
Replaces legacy JSON-in-Binary parameter envelopes with a length-prefixed, per-parameter binary format for all client→server calls. Introduces SerializeParametersToBinary and DeserializeParametersFromBinary for type-safe, zero-copy parameter handling. Marks IdMessage, SignalPostJsonDataMessage, and related wrappers as obsolete. Updates all client CRUD/messaging helpers and server-side dispatch to use object[] binary serialization. Adds DataSerializerType to SignalReceiveParams for response format indication. Updates tests and documentation to reflect new protocol. BREAKING CHANGE: not compatible with previous JSON-in-Binary clients/servers.
Major protocol update: OnReceiveMessage now takes metadata (SignalReceiveParams) and payload (byte[]) as separate hub arguments, not a single envelope. Metadata is AcBinary-serialized; payload uses protocol fast-path. Updated all client/server code, interfaces, and docs. Added ISignalParams and SignalReceiveParams types. Improved AcBinaryHubProtocol diagnostics and made byte[] fast-path more robust. This enables clearer, more debuggable, and future-proof SignalR binary messaging.
Removed OnReceiveStreamMessage from both AcSignalRClientBase and IAcSignalRHubBase, eliminating client-to-server streaming support. Updated StreamAsync calls to use the explicit method name string. This change enforces server-to-client streaming only and cleans up related interface and method references.
- Cache wrapper/metadata in serialization bridge methods to avoid redundant GetType and GetWrapper calls, improving performance.
- Update source generator to combine null/depth checks and cache depthExceeded for collections.
- Add useAcBinaryProtocol option to AcSignalRClientBase, allowing binary protocol to be toggled via constructor and registered via DI.
- Update documentation to reflect new caching rules and performance improvements.
Refactored AcBinarySerializer to add a source-generated (SGen) root fast path, bypassing the full runtime dispatch chain for SGen-decorated types and improving performance. Introduced helper methods for context pooling and buffer management. All entry points now use these helpers for consistency. Added comprehensive SGen architecture documentation (BINARY_SGEN.md), updated all related docs to explain the hybrid model, bridge methods, and configuration. Clarified doc loading rules in copilot-instructions.md for strict doc-first enforcement.
- Add TypeName to TestDataSet for clearer test scenario reporting
- Display serializer options in console and log outputs
- Extend BenchmarkResult with OptionsDescription
- Serializer benchmarks now provide detailed config summaries
- Log files now include test type and serializer options summary
- Generate .LLM Markdown results for LLM consumption and docs
- Reference .LLM results in BINARY_IMPLEMENTATION.md for visibility
Major documentation overhaul for binary serialization and SignalR:
- Simplified and clarified AyCode.Core README, with direct links to new deep-dive docs (BINARY_IMPLEMENTATION.md, BINARY_WRITERS.md)
- Added BINARY_WRITERS.md: detailed design and rationale for ArrayBinaryOutput and BufferWriterBinaryOutput, chunk sizing, and buffer management
- Refined BINARY_IMPLEMENTATION.md: clearer buffer management, output strategies, and hot-path rules; references new writers doc
- Added SIGNALR_BINARY_PROTOCOL.md: full wire format, zero-copy pipeline, dual BWO pattern, and read path for custom SignalR protocol
- Updated SignalRs README and SIGNALR.md: clarified protocol, tag system, request/response flow, and technical debt
- Improved cross-linking and discoverability throughout
These changes make the technical documentation clearer, more maintainable, and easier to navigate for advanced contributors.
- Add BINARY_IMPLEMENTATION.md with internal architecture and perf details
- Link new implementation doc from all relevant documentation
- Change default ReferenceHandlingMode to OnlyId
- Change default UseStringInterning to All
- Add InternalsVisibleTo for Mango.Nop.Core
- Added "AI AGENT CORE PROTOCOL" to all copilot-instructions.md files: mandates [LOADED_DOCS] prefix, hard-gates tool usage, enforces no-re-read of .md files, and requires user consent for doc/code changes.
- Updated CLAUDE.md to require reading copilot-instructions.md first.
- Added topic-based doc separation and folder navigation rules.
- Changed doc sync: agent now passively detects discrepancies and asks before updating docs.
- Every code-modifying response must end with a [DOCUMENTATION CHECK] section.
- Centralized measurement system and domain traps in new FruitBank.Common/docs/GLOSSARY.md; updated references in FruitBankHybridApp GLOSSARY.md.
- Clarified schema and doc locations in FruitBankHybridApp README.md.
- Added hybrid execution model section to AyCode.Core BINARY_FEATURES.md.
- Removed unnecessary BeginUpdate/EndUpdate calls in MgGridBase.cs for layout persistence.
- Removed full Toon schema from plugin SCHEMA.md to avoid duplication.
- CLAUDE.md: reduced to single-line pointer to copilot-instructions.md (eliminates redundant auto-loaded content)
- copilot-instructions.md: added @repo name field, relative paths in own-dep-repos, "do not re-read .md files" rule, and explicit permission to navigate external repos
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Refined guidance on avoiding redundant code by focusing on the current context rather than broad searches. Updated .md file sync instructions to prioritize code as the source of truth, clarify when and how to update documentation, and outline review practices. Improved wording for clarity and alignment with actual workflows.
Added standardized metadata blocks to the top of each project and repo documentation file. These blocks define project type, and for repos, include layer and dependencies, improving clarity and enabling better tooling support.
Replaced Markdown links with plain code-style paths in all README.md files for consistency. Updated references to logging, SignalR, and dynamic method dispatch documentation. Clarified some documentation paths and improved consistency in context/architecture sections. No code changes—documentation only.
Replaced all Markdown link syntax in documentation with inline code formatting and consistent relative paths. Updated all cross-references and tables for clarity and uniformity. No code or logic changes; docs only.
- Move all major feature docs (logging, binary, SignalR, DataSource) into per-project `docs/` folders with dedicated Markdown files.
- Split monolithic docs into focused files: `BINARY_FORMAT.md`, `BINARY_FEATURES.md`, `BINARY_OPTIONS.md`, `LOGGING.md`, `LOGGING_SERVER.md`, `LOGGING_REMOTE.md`, `SIGNALR.md`, `SIGNALR_SERVER.md`, `SIGNALR_DATASOURCE.md`.
- Update all references in `README.md`, `copilot-instructions.md`, `GLOSSARY.md`, and `ARCHITECTURE.md` to point to the correct per-project doc.
- Add documentation tables to each project’s `README.md` and clarify folder structure.
- Update `.csproj` files to include `docs/**/*.md` for packaging.
- Remove obsolete/moved docs from the solution file.
- Ensure all technical debt warnings and cross-references are preserved and accurate.
- No code changes; documentation only.
- Added SIGNALR.md (transport) and SIGNALR_DATASOURCE.md (collection) as layered, comprehensive documentation; retired SIGNALR_ARCHITECTURE.md
- Updated all .md files and READMEs to reference new docs and clarify separation between transport and DataSource
- Clarified CRUD tag structure (5 independent tags), single-method tag-based dispatch, and JSON-in-Binary tech debt
- Added slot allocation and wire format clarifications to serialization docs
- Improved documentation layering, conventions, and critical warnings for future maintainers
Added comprehensive docs for SignalR tag-based dispatch (docs/SIGNALR_ARCHITECTURE.md), including message flow, tag system, dynamic method registry, and tech debt (JSON-in-Binary). Updated all related READMEs, glossaries, and conventions to reference this architecture and clarify grid infrastructure (MgGridBase, FruitBankGridBase) and external dependency locations (AyCode.Core, AyCode.Blazor, Mango.Nop Libraries, FruitBank Plugin). Synchronized solution items and copilot-instructions. Improves discoverability, enforces conventions, and clarifies tech debt for all developers.
- Add BINARY_FORMAT.md: full AcBinary wire format spec (markers, encoding, options, protocol, interactions)
- Reference BINARY_FORMAT.md from GLOSSARY.md, Binaries/README.md, and Serializers/Binaries/README.md; add new glossary terms
- Clarify and expand config options tables to match new doc
- Add/clarify LLM maintenance rules: always sync .md files with code, auto-fix discrepancies
- Update root README.md: AyCode.Core targets .NET 9, not 10; stress doc/code sync
- Add code reuse and doc sync conventions to copilot-instructions.md and CONVENTIONS.md
- Add docs/ folder and BINARY_FORMAT.md to solution as Solution Items
- Minor clarifications and cross-links in ARCHITECTURE.md and other docs
- Introduced `.github/copilot-instructions.md` as the single source of truth for domain rules, conventions, and pitfalls in each solution.
- Added `CLAUDE.md` to guide Claude to read domain rules, glossary, and README before code generation.
- Updated all solution and project `README.md` files to document project purpose, structure, key files, and LLM context (Copilot/Claude/Cursor).
- Added or revised `docs/ARCHITECTURE.md`, `docs/CONVENTIONS.md`, and `docs/GLOSSARY.md` to clarify dependency graphs, naming, patterns, and terminology.
- For FruitBankHybridApp, added `docs/SCHEMA.md` (Toon format) and expanded the glossary with business/measurement terms and common traps.
- Updated all subfolder READMEs to list key files, conventions, and LLM maintenance notes.
- Ensured all documentation is cross-referenced, up-to-date, and includes explicit instructions for LLMs to keep docs in sync with code and avoid suggesting removal/rollback as a solution.
- Standardized documentation and onboarding for maintainability and LLM/code quality across all solutions.
Added comprehensive README.md documentation to every project and subfolder in the solution. Each README describes the purpose, key files, structure, dependencies, and design patterns for its area. This improves codebase navigability and maintainability, and includes a maintenance note to keep docs in sync with future code changes.
Added detailed README files for the root AyCode.Core library and each major subfolder (Compression, Consts, Enums, Extensions, Helpers, Interfaces, Loggers). Each README outlines folder purpose, key files, APIs, and critical notes, with tables and architecture diagrams where relevant. Included explicit instructions for LLM-based maintenance. This greatly improves project documentation and developer onboarding.
Updated test and benchmark project dependencies to latest versions (BenchmarkDotNet, coverlet.collector, MSTest, etc.). Reformatted source generator project references for consistency. Commented out most AcBinary benchmark variants in Program.cs, leaving only FastMode+Default enabled. Significantly expanded README with detailed architecture, file descriptions, and configuration for each serializer subfolder. Added LLM maintenance notes to ensure documentation stays in sync with code changes.
Added HasScanWork and related methods to determine at compile time if a property requires scan work. EmitScanProp now skips code generation for properties proven to not need scan logic, improving efficiency and reducing unnecessary code output.
Major refactor of binary serialization codegen and runtime:
- Added property writer bridge methods for markerless/metadata paths
- Centralized object marker logic via new bridge methods
- Simplified SGen output: single bridge call replaces branching
- FixObj slot markers now supported in serialization/deserialization
- Refactored collection/dictionary element serialization
- Removed redundant WritePropertyMarkerless method
- Improved tests: use BinaryTypeCode constants, FixObj parsing
- Added InternalsVisibleTo for test project access
- Annotated TestSimpleClass for SGen support
Reduces generated code size, improves maintainability, and ensures correct handling of new binary format features.
Split WriteObjectProperties into markerless and metadata variants for clarity and performance. Adjust method inlining attributes to favor hot path optimization. Comment out WritePropertyValue and some AcBinaryBenchmark variants to streamline code and benchmarks. Improves maintainability and serialization efficiency.
Split hot/cold paths for polymorphic and reference-tracked object serialization. Introduce dedicated cold-path methods for polymorphic and IId reference handling, and new helpers for writing reference indices. Extract property writing loop for reuse. Simplify WriteObject and update property dispatch logic to reduce branching and clarify marker handling.
Loretta