Benchmark results
This section contains the published Radixor benchmark reference set. It is intentionally split into three layers:
- benchmark reference pages, which explain methodology, corpora, environment, candidate selection, multilingual generalization, and the English coverage-speed deep dive;
- language result pages, which contain the actual same-language accuracy and throughput tables;
- pairwise quality pages and generated sections, which publish over-stemming, under-stemming, candidate-policy, classification, and partition measurements from one checked result snapshot.
This structure keeps methodology separate from per-language result pages, while preserving all measured data and the command-class analysis for each Radixor default model.
Read this first
Start with Benchmarking for the high-level interpretation model. Three separate dimensions must be read together: pairwise linguistic quality, exact-root agreement, and runtime. The principal quality benchmark tests whether forms in the same annotated dictionary group—a morphological family, not a semantic or synonym set—receive the same stem, while forms with no shared group membership remain separated. It does not require the resulting stem to equal one prescribed dictionary root; exact-root agreement is reported separately. Many competing stemmers are intentionally light, minimal, or aggressive, and can be fast because they perform a narrower or different linguistic transformation.
Radixor rows in the refreshed tables use contracted compiled patch tries. Contraction collapses uniform preferred-command subtrees into accepting leaves, reducing hot lookup depth while preserving the preferred result measured by the accuracy pass.
Radixor models were trained from the same lexical resources that define the pairwise reference relation. Pairwise results therefore measure same-resource agreement, not independent external linguistic validity. This limitation applies at the headline, language, and aggregate levels.
Reference pages
| Page | Purpose |
|---|---|
| Methodology | Workload design, normalization, speed metrics, quality metrics, and interpretation rules. |
| Linguistic quality methodology | Gold-standard groups, output policies, pairwise formulas, ranking rules, aggregation, and limitations. |
| Complete pairwise quality results | Lossless 286-row result matrix, placement tables, aggregate detail, secondary-metric leaders, and provenance. |
| Tested stemmers | Versions, upstream attribution, evaluated coverage, adapters, preprocessing, and output capability. |
| Reproducibility and raw data | Commands, versioned CSV snapshot, checksum, generated artifacts, and unavailable provenance. |
| Corpora | Dictionary row counts, complete quality tokens, already-root tokens, changed speed tokens, and timing token counts. |
| Environment and reports | Hardware, JVM, JMH settings, report files, and badge/report policy. |
| Dictionary-family generalization | All-language, five-split held-out-family results from 10% through 100% Java model training coverage. |
| Complete generalization results | Lossless 143-language × 10-level tables, including 100% rows and five-seed ranges. |
| Generalization methodology | Frozen nested splits, unseen-surface leakage control, formulas, reproduction, and limitations. |
| Edit-cost sensitivity | Language-specific edit-cost recommendations, exact command-equivalence classes, trie structure, and generalization associations. |
| Edit-cost methodology | Normalized cost grid, exact equivalence verification, frozen splits, outcomes, analysis rules, and reproduction. |
| English dictionary coverage | Quality/speed operating curve for contracted Radixor tries built from 100% down to 10% of English dictionary rows. |
| Candidate evaluation | Included benchmark families and evaluated candidates that were skipped. |
Language results
Each language page contains:
- the dictionary corpus size,
- the Radixor patch-command distribution,
- exact-root quality metrics,
- throughput metrics,
- pairwise linguistic-quality metrics showing whether same-group forms share a stem and forms with no shared dictionary-group membership remain separated,
- for the historical 20-language cohort only, the separately retained edit-cost and dictionary-knowledge analysis,
- interpretation notes for the compared stemmers.
Open Language Benchmark Pages for the complete language list.
Key published result
The English dictionary coverage benchmark shows the current contracted-trie operating curve. With
the full English dictionary, Radixor reaches 97.668% all-token exactness and 98.110%
changed-token exactness at 107.1 ns/token. Even with a deterministic 10% dictionary slice, it
keeps 93.057% all-token exactness and 77.327% changed-token exactness at 78.1 ns/token.
Those figures should not be reduced to a single speed badge. The evidence supports a quality/speed envelope: the amount and quality of dictionary knowledge affect stemming precision, while contracted tries reduce lookup cost in uniform regions of the compiled graph.
Quality versus performance
Each language page keeps exact-root accuracy, JMH latency, and pairwise linguistic-quality results in separate tables. No undocumented scalar combines them. The active 2026-09-11 Java corpus, exact-root, and speed reports generate the performance sections. The active 2026-09-13 pairwise-quality snapshot preserves the earlier quality rows and adds the separately validated six-language deterministic continuation. Snowball 3.1.0 rows appear only where the implementation exactly matches the model language. Readers should inspect quality and speed side by side; no cross-language Pareto ranking is inferred from workloads with different dictionaries and token counts.
New Snowball 3.1.0 rows
| New direct stemmer | All exact | Changed exact | Root preserved | Speed | Relative to same-language Radixor |
|---|---|---|---|---|---|
| Czech | 19.865% | 18.186% | 27.645% | 99.2 ns/token | 1.086× |
| Persian | 3.660% | 0.000% | 100.000% | 311.9 ns/token | 4.282× |
| Polish | 22.315% | 20.225% | 34.078% | 106.3 ns/token | 0.982× |
These rows describe exact agreement with each Radixor model dictionary and the measured direct API workload; they are not a universal linguistic ranking. All three Snowball rows have lower exact-root agreement than their same-language Radixor baseline. The Czech and Polish runtime intervals overlap their Radixor intervals and support no directional speed conclusion; the Persian intervals do not overlap and the Snowball point estimate is higher. Lucene 10.5.0 does not expose the three new algorithms through SnowballFilter, so no synthetic Lucene wrapper rows were added.
New Lucene 10.5.0 same-language rows
The 2026-09-13 bounded continuation added six real Lucene language filters that had previously
been missing from the same-language matrix. Bulgarian belongs to this group: Lucene provides
BulgarianStemFilter, while official Snowball 3.1.0 has no Bulgarian algorithm. The exact-root
counters are deterministic for the fixed dictionaries; runtime comes from a separate Ryzen 5
7600 campaign and is compared only within each contemporaneously measured pair.
| Language and Lucene filter | All exact | Changed exact | Root preserved |
|---|---|---|---|
| Arabic — ArabicStemFilter | 0.275% | 0.243% | 1.373% |
| Bulgarian — BulgarianStemFilter | 8.735% | 6.818% | 43.307% |
| Bengali — BengaliStemFilter | 3.853% | 3.243% | 18.919% |
| Greek — GreekStemFilter | 0.014% | 0.004% | 0.068% |
| Indonesian — IndonesianStemFilter | 80.839% | 78.888% | 89.605% |
| Latvian — LatvianStemFilter | 2.018% | 1.466% | 7.585% |
| Language | Radixor ms/op | Lucene ms/op | Lucene / Radixor |
|---|---|---|---|
| Arabic | 54.644 ± 1.813 | 55.670 ± 0.852 | 1.019× |
| Bulgarian | 3.040 ± 0.124 | 2.923 ± 0.098 | 0.962× |
| Bengali | 0.210 ± 0.003 | 1.566 ± 0.029 | 7.443× |
| Greek | 6.004 ± 0.179 | 15.250 ± 0.344 | 2.540× |
| Indonesian | 1.209 ± 0.018 | 1.636 ± 0.050 | 1.353× |
| Latvian | 5.087 ± 0.089 | 9.632 ± 0.172 | 1.894× |
The ratio is Lucene / Radixor; below 1 means the Lucene point estimate is lower. JMH errors are
99.9% intervals, and interval overlap is not a formal significance test. Download the
exact-root counters and
paired speed measurements, or inspect the
campaign environment and provenance.
Pairwise quality findings
The active snapshot covers all 143 default-model languages. Radixor's models were trained from the same lexical resources that define the benchmark's dictionary-reference relations. These results therefore measure same-resource agreement, not independent external linguistic validity. Third-party adapters receive the same evaluated forms but were not trained by this benchmark.
Direct comparison is available in 66 language-mode matrices across 33 unique languages (33 in ALL_WORDS; 33 in LOWERCASE_GROUPS_ONLY). The mode counts remain explicit because coverage is evaluated per mode and need not be identical.
Evidence-based primary-output result
Across the directly comparable cohort, Radixor achieved the highest balanced accuracy among the evaluated deterministic stemmers for all 33 languages across 66 directly compared matrices, with no exact first-place ties. 216 additional matrices contain only Radixor and establish no comparative rank; 4 matrices have undefined balanced accuracy. The claim is limited to the evaluated implementations, versions, dictionaries, adapters, and balanced-accuracy metric.
Full-coverage Radixor macro result
Macro balanced accuracy gives each language equal weight. It is a Radixor-only full-coverage aggregate, not a cross-stemmer common-language ranking.
| Dictionary mode | Defined languages | Macro balanced accuracy |
|---|---|---|
| ALL_WORDS | 141 / 143 | 0.984680 |
| LOWERCASE_GROUPS_ONLY | 141 / 143 | 0.985130 |
Balanced-accuracy leadership does not imply leadership on every error trade-off. The lossless appendix preserves every non-Radixor secondary-metric leader, all 286 language-mode matrix rows, placement tables, micro aggregates, and candidate-cohort caveats.
- Complete pairwise quality results
- Machine-readable quality snapshot
- SHA-256:
d41e00160cda44758e806c37d210f2d9b90b1ebad22eebe5ece89a40f4ed9ab0 - Linguistic quality methodology
Java Radixor and Snowball runtime comparison
The 2026-09-11 Java JMH snapshot charts all 29 supported official direct Snowball exact-language cases against the exact generic Radixor model row. The geometric mean is 2.104× over all 29 comparable changed-token cases. A ratio above 1 means Snowball used more time per token; ratios are point estimates, not categorical winners. The reported 99.9% JMH intervals overlap in 7 cases, so this summary does not support a directional conclusion for those cases. Non-overlap is not presented as a formal significance test. All rows use 3 warmup iterations, 5 measurement iterations, 3 forks, one thread, and the same per-language timing population.
Accessible raw Java comparison table
| Language | Model | Timing corpus | Radixor ns/token | Radixor JMH error | Snowball ns/token | Snowball JMH error | Snowball / Radixor | 99.9% intervals | Aggregate |
|---|---|---|---|---|---|---|---|---|---|
| Arabic | ar-default | changed tokens | 159.8 | ±33.8 | 444.1 | ±18.7 | 2.780× | do not overlap — not a formal significance test | yes |
| Armenian | hy-am-default | changed tokens | 88.9 | ±19.6 | 146.2 | ±19.2 | 1.644× | do not overlap — not a formal significance test | yes |
| Catalan | ca-es-default | changed tokens | 85.9 | ±12.6 | 248.5 | ±12.4 | 2.895× | do not overlap — not a formal significance test | yes |
| Czech | cs-cz-default | changed tokens | 91.4 | ±16.5 | 99.2 | ±15.8 | 1.086× | overlap — no directional conclusion | yes |
| Danish | da-dk-default | changed tokens | 59.3 | ±9.0 | 120.7 | ±13.1 | 2.035× | do not overlap — not a formal significance test | yes |
| Dutch | nl-nl-default | changed tokens | 78.4 | ±14.1 | 203.2 | ±15.1 | 2.592× | do not overlap — not a formal significance test | yes |
| Estonian | et-ee-default | changed tokens | 61.5 | ±12.6 | 195.9 | ±13.3 | 3.185× | do not overlap — not a formal significance test | yes |
| English | us-uk-default | changed tokens | 131.3 | ±9.3 | 229.0 | ±4.7 | 1.745× | do not overlap — not a formal significance test | yes |
| Finnish | fi-fi-default | changed tokens | 181.9 | ±21.2 | 172.8 | ±8.7 | 0.950× | overlap — no directional conclusion | yes |
| French | fr-fr-default | changed tokens | 162.1 | ±4.9 | 320.9 | ±17.5 | 1.980× | do not overlap — not a formal significance test | yes |
| German | de-de-default | changed tokens | 189.2 | ±16.4 | 410.1 | ±15.8 | 2.167× | do not overlap — not a formal significance test | yes |
| Greek | el-gr-default | changed tokens | 121.0 | ±24.2 | 740.5 | ±28.7 | 6.119× | do not overlap — not a formal significance test | yes |
| Hungarian | hu-hu-default | changed tokens | 73.5 | ±10.2 | 193.5 | ±17.2 | 2.634× | do not overlap — not a formal significance test | yes |
| Indonesian | id-id-default | changed tokens | 92.0 | ±12.1 | 108.4 | ±12.1 | 1.178× | overlap — no directional conclusion | yes |
| Irish | ga-ie-default | changed tokens | 103.7 | ±12.8 | 139.0 | ±13.4 | 1.340× | do not overlap — not a formal significance test | yes |
| Italian | it-it-default | changed tokens | 97.6 | ±19.4 | 369.7 | ±16.6 | 3.788× | do not overlap — not a formal significance test | yes |
| Lithuanian | lt-lt-default | changed tokens | 128.7 | ±16.7 | 153.7 | ±12.5 | 1.194× | overlap — no directional conclusion | yes |
| Norwegian Bokmal | nb-no-default | changed tokens | 71.2 | ±10.5 | 98.5 | ±12.6 | 1.383× | do not overlap — not a formal significance test | yes |
| Norwegian Nynorsk | nn-no-default | changed tokens | 54.2 | ±10.0 | 96.7 | ±12.9 | 1.785× | do not overlap — not a formal significance test | yes |
| Persian | fa-ir-default | changed tokens | 72.8 | ±22.8 | 311.9 | ±20.3 | 4.282× | do not overlap — not a formal significance test | yes |
| Polish | pl-pl-unimorph | changed tokens | 108.3 | ±19.6 | 106.3 | ±17.1 | 0.982× | overlap — no directional conclusion | yes |
| Portuguese | pt-pt-default | changed tokens | 68.5 | ±16.3 | 266.2 | ±13.1 | 3.888× | do not overlap — not a formal significance test | yes |
| Russian | ru-ru-default | changed tokens | 142.2 | ±28.0 | 174.7 | ±16.1 | 1.228× | overlap — no directional conclusion | yes |
| Romanian | ro-ro-default | changed tokens | 93.6 | ±13.3 | 329.2 | ±16.3 | 3.517× | do not overlap — not a formal significance test | yes |
| Southern Sotho | st-za-default | changed tokens | 50.1 | ±12.9 | 53.1 | ±12.3 | 1.061× | overlap — no directional conclusion | yes |
| Spanish | es-es-default | changed tokens | 106.3 | ±18.8 | 229.9 | ±16.7 | 2.164× | do not overlap — not a formal significance test | yes |
| Swedish | sv-se-default | changed tokens | 77.4 | ±10.1 | 103.7 | ±12.8 | 1.340× | do not overlap — not a formal significance test | yes |
| Turkish | tr-tr-default | changed tokens | 117.4 | ±13.1 | 442.0 | ±19.3 | 3.765× | do not overlap — not a formal significance test | yes |
| Yiddish | yi-default | changed tokens | 55.0 | ±17.0 | 320.4 | ±21.0 | 5.829× | do not overlap — not a formal significance test | yes |