Pattern generation metrics ========================== Seal5 records measurements for each PatternGen invocation in ``.td.stats``. The existing LLVM statistics groups are preserved; additional measurements live under ``seal5``. This works without LLVM statistics support, including for failed invocations and format-only generation. The PatternGen writer automatically emits ``pattern_gen_manifest.json``, ``pattern_gen_report.json`` and ``pattern_gen_report.csv`` in each model's temporary output directory. The manifest records current-run instructions, including skipped and failed instructions. The reports contain instruction, set and model-global rows. Numeric totals also enter the existing pass metrics. For a report across all models in an existing workspace:: python -m seal5.backends.report.pattern_gen.writer \ --yaml /path/to/.seal5/settings.yml \ --scope all --output pattern-metrics.json Use ``--scope instruction``, ``set`` or ``global`` to restrict the report. ``--agg`` is an alias for ``--scope global``. CSV, Markdown and pickle output remain supported. Aggregated output now contains one row per aggregation scope, rather than the old two-column raw LLVM statistic sums. Metric definitions ------------------ * ``n_sets`` counts distinct (model, set) identities, including sets containing only skipped instructions. ``n_processed_sets`` counts sets with at least one attempted invocation. Parsed-set counters from separate subprocesses are not added to estimate the number of sets. * ``n_instructions`` counts instructions considered by the writer; ``n_processed_instructions`` excludes skipped instructions. ``n_success``, ``n_failed`` and ``n_skipped`` describe instruction outcomes, not pattern counts. * ``n_extracted_patterns`` is the number of generated rules/entries, using the PatternGen counter when present, otherwise its ``Pattern for`` output records. One instruction can produce multiple patterns. Extraction does not establish importability, test coverage or semantic correctness. * ``n_tablegen_patterns`` counts ordinary TableGen patterns; ``n_gisel_table_patterns`` counts extracted roots for direct GISel generation. ``n_gisel_table_entries`` counts the expanded RuleMatcher entries, including permutations of commutative operands. For example, the scalar NAND example extracts one pattern but emits four entries. Entry counting recognizes the current emitter's literal permutation-loop syntax; unfamiliar control flow yields an unavailable count. Direct entries bypass the SelectionDAG-to-GlobalISel importer. * ``pattern_gen_time_s`` sums subprocess elapsed times, measured with a monotonic clock. ``tablegen_time_s`` and ``gisel_table_time_s`` separate the two generation modes. They include parsing, optimization, extraction and output in that invocation, including format generation when requested. They exclude LLVM builds and Seal5 postprocessing. They are not internal phase timings. * ``generation_wall_time_s`` in the writer's global report/pass metrics measures the generation phase including Seal5 postprocessing, excluding model loading and report serialization. Parallel subprocess time sums can exceed this wall time; do not interpret them as CPU time. * ``tablegen_bytes`` counts UTF-8 file bytes in the final per-instruction ``.td`` output plus its instruction-format ``.td`` when requested. This includes final feature predicates and pattern-priority adjustments, but excludes shared include scaffolding. Direct GISel mode can also generate TableGen definitions; these bytes are kept separate from its matcher table. * ``gisel_table_bytes`` counts the final generated ``.inc`` files. ``gisel_table_patch_bytes`` counts corresponding ``.inc.patch`` sidecars: reproducible unified addition patches targeting ``llvm/utils/TableGen/seal5//.inc``. These include diff headers and line prefixes. They measure matcher-table payload patches, not the entire backend support patch or shared GlobalISelEmitter include changes. Seal5 still installs the table through its existing File artifact; the sidecar is for measurement/reproduction, not an additional patch to apply in the flow. Sizes count generated artifacts even if an invocation subsequently fails; absent files have size zero. Skipped instructions contribute zero. Failed subprocesses retain their timing and return code. A new invocation removes old output artifacts before measuring. Current-run manifests prevent stale statistics from skipped instructions from re-entering the report. Import and match evidence ------------------------- The reports distinguish ``n_tablegen_patterns_imported``, ``n_tablegen_patterns_matched`` and ``n_gisel_table_patterns_matched`` and ``n_gisel_table_entries_matched``. Importing means acceptance by the GlobalISel importer, not selection at runtime. Matching means a distinct generated rule/entry was selected in codegen validation, not that the generated instruction has been proved semantically correct. Multiple executions of one rule count once. Direct entries have no importer count. Current PatternGen versions do not provide per-pattern import or match evidence. Their counts are therefore unavailable (JSON null / empty CSV cells), not zero and not the extraction count. Whole-backend ``gisel-emitter`` totals are retained in raw statistics but cannot be attributed to one generated instruction. If an instrumented PatternGen emits scoped ``Imported patterns`` / ``Matched patterns`` counters in its ``pattern-gen`` group, these are consumed directly. Alternatively supply ``--validation evidence.json``. This is a JSON list, with one entry per (model, set, instruction, mode):: [ { "model": "Example", "set": "XExample", "instruction": "ADD3", "mode": "tablegen", "imported": [0, 1], "imported_source": "artifacts/import-records.json", "matched": [1], "matched_source": "artifacts/rule-coverage.json" } ] Indices are zero-based in the generator's pattern-output order. A supplied list is a complete result for that invocation: an empty list means zero validated patterns, whereas an omitted list means unavailable. The source field identifies the evidence to retain with the experiment; the reporter validates identities, index ranges and duplicates, but does not independently verify that evidence. Use ``mode: gisel_table`` for direct generation: ``matched`` identifies extracted roots, while ``matched_entries`` / ``matched_entries_source`` can independently record expanded-entry coverage, with indices in emitted entry order. Do not supply ``imported`` for direct generation. A passing instruction-level FileCheck test cannot establish which of several alternative patterns matched: supply rule-level evidence rather than counting every pattern of the instruction as tested. An aggregate remains unavailable if any contributing instruction lacks that measurement. ``*_known`` reports the measured subtotal and ``*_unreported_instructions`` reports the missing population. This prevents a partial coverage result from appearing to be a complete total. Reports also accept older ``.td.stats`` files without a manifest. In this case instruction enumeration, timing, sizes and generation mode may be incomplete; regenerate the artifacts for a controlled experiment. Older files are assumed to use ordinary TableGen mode because they did not record the generation mode.