Skip to content

Benchmarking

This page is the entry point for benchmark interpretation. Detailed tables and long reference material are split into focused subpages so that important points do not get buried.

Three independent evidence streams

%%{init: {"flowchart": {"wrappingWidth": 320}}}%%
flowchart LR
    accTitle: Three independent benchmark evidence streams
    accDescr: Reference agreement, held-out generalization, and runtime timing each answer a different question, report a distinct result, and carry a separate limitation. The streams must be read together and never collapsed into one score.

    START["Read all three<br/>evidence streams"]
    START --> REF["Reference agreement<br/>Pairwise quality metrics<br/>Same-resource agreement"]
    START --> GEN["Held-out generalization<br/>Five-split exactness<br/>Within-resource transfer"]
    START --> RUN["Runtime timing<br/>JMH time and interval<br/>Campaign-specific evidence"]
    REF --> BOUNDARY["Interpret together<br/>Never collapse into one score"]
    GEN --> BOUNDARY
    RUN --> BOUNDARY

Radixor does not publish a single composite benchmark score. Reference agreement asks whether predicted group relationships match a dictionary-derived relation. Held-out generalization asks whether learned commands transfer to withheld families from that resource. Runtime timing measures execution cost on a declared machine and workload. Each stream has its own inputs, methods, limitations, and provenance; none substitutes for another.

Key takeaways

  • Speed and accuracy must be read together. A faster row is not necessarily a better stemmer.
  • Radixor ranks uniquely first by balanced accuracy in all 66 directly compared language-mode matrices across 33 unique languages: 33 in ALL_WORDS and 33 in LOWERCASE_GROUPS_ONLY. Another 216 matrices contain only a Radixor row and therefore establish no comparative rank; four matrices have undefined balanced accuracy. In the current Java speed matrix Radixor records the lowest runtime point estimate in 18 of the 30 languages with a direct comparator. Per-language tables remain authoritative because point estimates, uncertainty, and linguistic objectives must be interpreted together.
  • The measured Radixor cost accompanies higher dictionary-root agreement in the published matrix. Search-system relevance is deployment-specific and is not measured by these stemming benchmarks.
  • Radixor models were trained from the same lexical resources that define the pairwise dictionary-reference relation. The quality result is therefore a same-resource agreement result, not an independent external validation of linguistic correctness.
  • Speed benchmarks process changed dictionary tokens where the surface form differs from the expected root. Accuracy benchmarks process the complete dictionary.
  • Absolute timing varies by machine and shared-runner state. Compare ratios only within one declared campaign, reproduce deployment-sensitive results on the target hardware, and do not infer a guarantee from a one-off point estimate.
  • Accuracy tables use deterministic auxiliary counters from a single non-timed JMH evaluation, while Radixor counters are independently cross-checked by the default-model corpus report. Runtime scores from accuracy methods are not interpreted.
  • The historical Porter performance badge is retired. Benchmark reporting now uses speed and quality tables rather than a single Porter ratio.
  • Edit-cost effects are not assumed to transfer between languages. The published 20-language archive contains a 234,000-observation logical matrix. The current standard/default scope is the 31 models in the standard package. Campaigns from different source states or machines remain separate, and a non-baseline selection remains an externally unvalidated candidate.
  • Exact-language comparator continuations are published as separate campaigns. Deterministic quality counters can extend the accuracy evidence, while speed requires a same-hardware Radixor/comparator pair; timing rows from different machines are never pooled.

Benchmark suite boundaries

Radixor contains internal trie microbenchmarks, a separate stemmer comparison suite, and a dictionary coverage benchmark for Radixor itself. The current default-model publication uses the same-language speed and exact-root methods selected by the command recorded on the environment page. Internal FrequencyTrie* microbenchmarks, the optional PolishPolimorfStemmerComparisonBenchmark, and the separate German CISTEM gold-standard experiment are not part of these language tables.

Every current default Radixor benchmark scenario uses the model ID declared by its Language.defaultModelId(). The root JMH runtime configuration depends directly on all default model projects plus optional pl-pl-polimorf; no benchmark-pack project or artifact exists. These dependencies are benchmark-only and never enter the root published POM. A PoliMorf comparison is labeled with model ID pl-pl-polimorf, while the default Polish row remains pl-pl-unimorph.

The optional model has a verified complete compiled loading path. It is not included in the 2026-08-25 corpus, accuracy, speed, coverage, or stemming-quality measurements. Its documented startup heap is independent of ordinary runs, and any reported measurement requires the exact model artifact version and checksum.

Find the evidence

Page Purpose
Benchmark methodology Workload design, speed pass, quality pass, normalization policy, and exact-root metrics.
Linguistic quality methodology Pairwise gold standard, over/under-stemming, candidate policies, metrics, and ranking rules.
Complete pairwise quality results Lossless 286-row language-mode matrix, placement tables, aggregates, secondary-metric leaders, raw-data links, and provenance.
Tested stemmers Upstream attribution, tested versions, language coverage, adapter behavior, and limitations.
Reproducibility and raw data Versioned quality snapshot, checksum, commands, reports, and provenance limitations.
Benchmark corpora Dictionary row counts, complete quality tokens, already-root tokens, changed speed tokens, and timing token counts.
Benchmark environment and reports Hardware, OS, JVM, JMH settings, report files, and current badge/report policy.
Dictionary-family generalization Five-split, all-language evaluation of transformations applied to families withheld from Java model training.
Complete generalization results Lossless 143-language × 10-level appendix with five-seed medians and ranges, including 100% rows.
Generalization methodology Frozen sampling protocol, unseen-form leakage control, metrics, raw counters, and limitations.
Edit-cost sensitivity Normalized edit-cost choices, exact command-equivalence classes, trie structure, and held-out-family quality.
Edit-cost methodology Frozen cost grid, full-dictionary equivalence proof, stratified associations, recommendation rule, and limitations.
English dictionary coverage benchmark The quality/speed operating curve for contracted Radixor tries built from 100% down to 10% of English dictionary rows.
Candidate evaluation Included benchmark families and evaluated candidates that were skipped.
Language benchmark pages Per-language accuracy, speed, generalization, edit-cost evidence, factor analysis, and conclusions.

How to read results

Start with the language benchmark pages. Each language page lists accuracy first and speed second because throughput without root agreement is not enough to interpret stemmer quality.

Read the accuracy and runtime tables together. Lucene light/minimal filters and possessive filters intentionally perform narrower transformations; the published tables preserve those rows rather than treating different linguistic objectives as equivalent.

The all-language generalization benchmark isolates forms from withheld dictionary families across five frozen splits. The separate English dictionary coverage benchmark retains the original whole-dictionary quality/speed operating curve. Together they distinguish transfer evidence from the model-size/runtime trade-off.

The edit-cost experiment then tests 234 normalized relative-cost settings at every knowledge level. Its macro report identifies cross-language structure; the linked language sections state the evidence-derived conclusion separately for each dictionary and expose when a correlation lacks complete 45-stratum support.

Current result locations

The current measured language results are published in Language Benchmark Pages. Generated local report files for this benchmark update are listed in Benchmark environment and reports.

JMH TXT and CSV reports are still published as benchmark artifacts. They are no longer converted into a Shields endpoint benchmark badge. Model IDs, independent artifact versions, and descriptor checksums identify the inputs in the checked corpus snapshot. The optional PoliMorf model must not be attributed to the default Polish results. See Model Selection and Loading and Reproducibility.