31 lines
3.7 KiB
Markdown
31 lines
3.7 KiB
Markdown
# Reporting
|
||
|
||
Shared reporting types — the `BenchmarkResult` DTO that captures one cell of a benchmark run + the `BenchmarkReportWriter` that turns a list of these into the unified `.log` / `.LLM` / `.output` triplet + the `ReportingContext` bundle that parameterises both runners.
|
||
|
||
## Layout
|
||
|
||
- [`BenchmarkResult.cs`](BenchmarkResult.cs) — per-cell result row. `(TestData × Engine × IoMode × OptionsPreset × DispatchMode)` tuple + Ser / Des / RT timings (median, min, max, stddev — all ms-batch units), iter counts (post-calibration), allocated bytes per op, setup-side one-time alloc, `IsRoundTripOnly` flag, derived `SerializerName`. Pure DTO — no behaviour. Populated by either:
|
||
- Console `BenchmarkLoop.RunBenchmarksForTestData` (after adaptive measurement)
|
||
- BDN `BdnSummaryAdapter.Translate` (after BDN Summary is in hand)
|
||
|
||
- [`ReportingContext.cs`](ReportingContext.cs) — record bundle for the writer:
|
||
- `SourceTag` — `"Console"` / `"Bdn"`; drives the filename prefix
|
||
- `ResultsDirectory` — resolved at startup via `ResolveResultsDirectory()` walking up from `AppContext.BaseDirectory` to the nearest `AyCode.Core.sln`, then `Test_Benchmark_Results/Benchmark/`. Worktree-aware.
|
||
- `BuildConfiguration` — `"Debug"` / `"Release"` / `"NativeAOT"`; rendered into both the filename AND the report header
|
||
- `Utf8NoBom` — shared `UTF8Encoding(false)` for all `File.WriteAllText` calls
|
||
- `CharsetName`, `WarmupIterations`, `BenchmarkSamples`, `TargetSampleMs`, `UnstableCVThreshold` — run-header info embedded in every emitted artifact
|
||
|
||
- [`BenchmarkReportWriter.cs`](BenchmarkReportWriter.cs) — the writer itself:
|
||
- `SaveAll(ctx, results, testDataSets)` — orchestrator. Writes the `.log` (formatted text + CSV + per-cell tables + Overall aggregation), `.LLM` (markdown table + Overall aggregation), and `.output` (hex dump of the Large cell's AcBinary serialization). All three land in `ctx.ResultsDirectory` with the `{ctx.SourceTag}.FullBenchmark_{Build}_{ts}.<ext>` filename pattern.
|
||
- `PrintGroupedResults(results, testDataSets)` — colored per-cell tables to `System.Console`. Highlights MemoryPack (baseline) and AcBinary (SGen-Byte[]) rows with green/red win/lose colors, footer row shows pct deltas per metric.
|
||
- `PrintResult(result)` — single-line summary printed during the per-cell loop (real-time progress signal).
|
||
- `ComputeOverallStats(acResults, mpResults, valueSelector)` — paired-cell aggregation across `TestDataName` (arithmetic mean / geometric mean / median of per-cell ratios). Null-safe.
|
||
- `FormatMicrosWithRange(...)` — `26.86 (24.50..29.10)` style with ⚠️CV-warning suffix when stddev/median exceeds the `UnstableCVThreshold`. All formatting goes through `CultureInfo.InvariantCulture` so the CSV section in `.log` stays parseable regardless of the host locale.
|
||
- `ToPerOpMicros` / `SerPerOp` / `DesPerOp` / `RtPerOp` / `ToKilobytes` / `FormatPctSigned` / `FormatHexDump` / `AppendOverallLine` — helper utilities used inline by the report-rendering methods.
|
||
|
||
## Conventions
|
||
|
||
- **Time units in `BenchmarkResult`**: all `*TimeMs` fields are total-batch milliseconds. Per-op µs = `TimeMs / Iterations * 1000`. For BDN-sourced rows the adapter stores `Mean_ns / 1e6` with `Iterations = 1`, so the same formula yields per-op µs directly (`ms * 1000 = µs`).
|
||
- **InvariantCulture** is enforced everywhere a numeric value is rendered to file (`.log` CSV section, `.LLM` markdown cells). Console-output (the colored tables) uses default culture for human-friendliness.
|
||
- **`SourceTag` discriminator**: appears in three places — the filename prefix (`Console.` / `Bdn.`), the `.log` header (`║ Source: Console`), the `.LLM` H1 (`# AcBinary Benchmark [Console] Release ...`). Anyone diffing or grepping outputs can pin the source unambiguously.
|