Irish stemmer benchmarks
This page reports dictionary corpus, exact-root agreement, and runtime evidence for the independently available ga-ie-default Irish model.
Dictionary size: . The exact count is 24,035 distinct usable word forms after parser-compatible filtering and exact, case-preserved deduplication. Stars rank dictionary size relative to all benchmarked dictionaries in five nearly equal groups; they do not measure linguistic quality or benchmark accuracy.
The language metadata declares left-to-right writing.
All speed values are environment-specific and were measured on the hardware and JVM listed in the benchmark overview. The command distribution, exact-root accuracy, and speed tables belong to the published 2026-09-11 Radixor/Java 4.4.0 snapshot. Speed benchmark operations process changed tokens. Accuracy uses the complete Radixor dictionary for the language.
How to read this page
Start with the corpus and patch-command distribution, then compare exact-root agreement with runtime. Dictionary-size stars are contextual metadata, not an accuracy result.
Dictionary corpus
| Model ID | Model version | Language | Dictionary rows | Distinct usable forms | Complete quality tokens | Already-root tokens | Changed tokens | Timing workload | JMH timing tokens |
|---|---|---|---|---|---|---|---|---|---|
ga-ie-default |
1.0.0 |
GA_IE |
7,287 | 24,035 | 24,533 | 7,287 | 17,246 | changed tokens | 17,246 |
Radixor patch-command distribution
Radixor stores the preferred transformation for each normalized dictionary word form as a compiled patch command. This distribution shows which runtime command class is selected by the trained trie for the complete default-model dictionary. The total number of preferred patch commands analyzed for this language is 24,533.
| Command class | Meaning | Word forms | Share |
|---|---|---|---|
AppendCharacterCommand |
Appends one character to the end of the word form. | 7 | 0.029% |
BackwardCompoundCommand |
Applies a multi-step backward patch made from skip, delete, insert, and replace operations. | 11,162 | 45.498% |
DeleteSuffixCommand |
Deletes one or more trailing characters from the word form. | 5,691 | 23.197% |
PreserveCommand |
Returns the word form unchanged because it already matches the preferred root. | 7,392 | 30.131% |
ReplaceLastCharacterCommand |
Replaces the final character of the word form. | 281 | 1.145% |
Accuracy
Accuracy uses the complete dictionary and reports exact agreement with the dictionary root for each identified model and candidate.
| Stemmer | All exact | Changed exact | Root preserved | Note |
|---|---|---|---|---|
| Radixor | 97.970% | 97.536% | 98.998% | Exact model-ID benchmark; measured in this snapshot. |
| Official Snowball direct (Java) | 24.449% | 2.575% | 76.218% | Official Snowball 3.1.0 generated Java stemmer; measured in this snapshot. |
| Lucene SnowballFilter | 24.449% | 2.575% | 76.218% | Lucene integration of the matching Snowball algorithm; measured in this snapshot. |
Speed
Speed uses JMH average time, 3 warmup iterations, 5 measurement iterations, 3 independent forks, and 1 thread.
The canonical timing workload uses changed tokens when available; a root-only dictionary uses its complete root-preservation corpus. Smaller populations are repeated deterministically to the timing minimum. Relative factors use the Radixor row as the baseline.
| Stemmer | Benchmark method | Score ms/op | Error ms | ns/token | Relative vs Radixor | Note |
|---|---|---|---|---|---|---|
| Radixor | radixor[ga-ie-default] |
1.789 | 0.220 | 103.7 | 1.000 | Canonical model timing workload; measured in this snapshot. |
| Official Snowball direct (Java) | snowballDirect[IRISH] |
2.397 | 0.231 | 139.0 | 1.340 | Official generated Java stemmer; measured in this snapshot. |
| Lucene SnowballFilter | luceneSnowballFilter[IRISH] |
3.101 | 0.297 | 179.8 | 1.733 | Lucene TokenStream integration; measured in this snapshot. |
Interpretation notes
- The star tier reflects only relative distinct-form count among all benchmarked dictionaries.
- Values shown above come from the identified canonical benchmark snapshot.
- Runtime and exact-root agreement describe different properties and must be interpreted together.
Dictionary-family generalization conclusion
This is the language-specific conclusion from the independent radixor-generalization-v1 baseline
experiment. It is intentionally separate from the wider edit-cost protocol below; values from
the two frozen snapshots are not substituted for one another.
Evidence
Model ga-ie-default version 1.0.0 is evaluated over five
predeclared nested splits. Unseen metrics remove withheld occurrences whose normalized surface
also appeared in training. Parentheses show the observed split minimum–maximum.
| Training rows | Median unseen occurrences | Unseen all exact | Unseen changed exact | Unseen root preserved |
|---|---|---|---|---|
| 10% | 21,938 | 52.789% (52.518–53.743) | 36.464% (35.603–37.611) | 91.687% (91.525–92.561) |
| 20% | 19,432 | 56.294% (55.272–56.642) | 41.141% (39.771–41.389) | 92.039% (91.943–92.453) |
| 30% | 16,925 | 57.339% (56.455–58.407) | 42.711% (41.117–43.849) | 92.463% (92.177–93.337) |
| 40% | 14,494 | 58.366% (57.960–60.425) | 43.582% (42.943–46.513) | 93.353% (92.730–93.718) |
| 50% | 12,029 | 58.873% (58.630–60.254) | 43.920% (43.837–46.118) | 93.630% (92.576–93.996) |
| 60% | 9,534 | 60.036% (58.677–61.097) | 45.376% (44.646–46.897) | 93.897% (93.196–94.281) |
| 70% | 7,180 | 59.959% (59.763–61.558) | 46.037% (45.003–47.023) | 94.747% (93.942–94.855) |
| 80% | 4,727 | 62.021% (59.558–62.780) | 47.969% (45.253–48.409) | 95.400% (94.433–95.593) |
| 90% | 2,441 | 62.515% (58.703–64.252) | 48.046% (45.339–50.654) | 95.286% (94.193–96.591) |
Generalization conclusion
- Median exactness on genuinely unseen changed forms moves from 36.464% at 10% training knowledge to 48.046% at 90%, a measured +11.582 percentage-point change.
- Unseen all-form exactness moves from 52.789% at 10% training knowledge to 62.515% at 90%, a measured +9.726 percentage-point change.
- Preservation of unseen already-root forms moves from 91.687% at 10% training knowledge to 95.286% at 90%, a measured +3.599 percentage-point change.
- The evidence establishes within-resource transfer across withheld dictionary families. It does not estimate unrelated domains, misspellings, arbitrary compounds, or external corpora.
The complete ten-level table and split ranges remain in the complete generalization appendix; raw counters and provenance are in active machine-readable snapshot. The frozen methodology defines family-level splitting, unseen-surface leakage control, aggregation, and the limits of the claim.
Stemming quality
Runtime performance and linguistic grouping quality are independent dimensions. This section evaluates language GA_IE using the complete validated stemming-quality result matrix and the canonical linguistic-quality methodology, including its overlapping-group relation, dictionary modes, output policies, metrics, and ranking rules.
Radixor's model was trained from the same lexical resource that defines this benchmark's dictionary-reference relation. The result therefore measures same-resource agreement, not independent external linguistic validity. Third-party adapters receive the same evaluated forms but were not trained by this benchmark.
The tables retain both canonical dictionary-processing modes and every applicable output policy without redefining them on each language page. Download the complete machine-readable result snapshot.
Evaluation scope and key findings
The default model is ga-ie-default, loaded from classpath resource org/egothor/stemmer/models/ga-ie-default/stemmer.gz. The following findings compare only deterministic PRIMARY_OUTPUT rows over identical included groups; candidate policies are reported separately as capability analyses.
- ALL_WORDS:
Radixorranks first by balanced accuracy at 0.975677 among 3 deterministic stemmers. The runner-up isSNOWBALL IRISH DIRECTat 0.512787, a difference of 0.462890. This rank does not imply leadership in throughput or every secondary metric. - LOWERCASE_GROUPS_ONLY:
Radixorranks first by balanced accuracy at 0.975677 among 3 deterministic stemmers. The runner-up isSNOWBALL IRISH DIRECTat 0.512787, a difference of 0.462890. This rank does not imply leadership in throughput or every secondary metric.
ALL_WORDS
This mode contains 5 result rows, 3 evaluated stemmers, and 3 output policies. Applied-row and form counts are shown per row because adapters share the language corpus but policy rows remain independently auditable. Where at least two deterministic rows are available, PRIMARY_OUTPUT and ALL_CANDIDATES rankings are ordered by unrounded balanced accuracy, followed by MCC, F1, over-stemming rate, over-stemming count, under-stemming rate, and stemmer. ANY_CANDIDATE has no single rank metric and is listed alphabetically. Balanced accuracy is a navigation metric, not a universally authoritative quality score.
PRIMARY_OUTPUT ranking
| Rank | Stemmer | Balanced accuracy | Over-stemming (OI) | Under-stemming (UI) |
|---|---|---|---|---|
| 1 | Radixor | 0.975677 | 0.000000% | 4.864647% |
| 2 | SNOWBALL IRISH DIRECT | 0.512787 | 0.000290% | 97.442280% |
| 3 | SNOWBALL IRISH LUCENE FILTER | 0.512787 | 0.000290% | 97.442280% |
Classification metrics
| Rank | Stemmer | Output policy | Precision | Recall | Specificity | Balanced accuracy | Pairwise accuracy | Error rate |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 1.000000 | 0.951354 | 1.000000 | 0.975677 | 0.999982 | 0.000018 |
| 2 | SNOWBALL IRISH DIRECT | PRIMARY_OUTPUT | 0.764888 | 0.025577 | 0.999997 | 0.512787 | 0.999638 | 0.000362 |
| 3 | SNOWBALL IRISH LUCENE FILTER | PRIMARY_OUTPUT | 0.764888 | 0.025577 | 0.999997 | 0.512787 | 0.999638 | 0.000362 |
Pair-relation metrics
| Rank | Stemmer | Output policy | F0.5 | F1 | F2 | Jaccard | Fowlkes–Mallows | MCC |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 0.989877 | 0.975070 | 0.960700 | 0.951354 | 0.975374 | 0.975365 |
| 2 | SNOWBALL IRISH DIRECT | PRIMARY_OUTPUT | 0.112798 | 0.049499 | 0.031706 | 0.025378 | 0.139870 | 0.139829 |
| 3 | SNOWBALL IRISH LUCENE FILTER | PRIMARY_OUTPUT | 0.112798 | 0.049499 | 0.031706 | 0.025378 | 0.139870 | 0.139829 |
Raw pair counts
| Rank | Stemmer | Output policy | TP | FP | FN | TN | Over error / possible | Under error / possible |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 101283 | 0 | 5179 | 288722133 | 0 / 288722133 | 5179 / 106462 |
| 2 | SNOWBALL IRISH DIRECT | PRIMARY_OUTPUT | 2723 | 837 | 103739 | 288721296 | 837 / 288722133 | 103739 / 106462 |
| 3 | SNOWBALL IRISH LUCENE FILTER | PRIMARY_OUTPUT | 2723 | 837 | 103739 | 288721296 | 837 / 288722133 | 103739 / 106462 |
ANY_CANDIDATE oracle bounds
These results are measured, not missing. ANY_CANDIDATE answers two separate optimistic questions for each pair: a gold-related pair avoids under-stemming when the candidate sets intersect, while a gold-negative pair avoids over-stemming when some non-colliding candidate selection exists. The oracle may choose a different candidate for the same word in different pairs. Consequently, these decisions do not form one globally realizable predicted relation or one TP/FP/FN/TN confusion matrix. Balanced accuracy, F-scores, Jaccard, Fowlkes–Mallows, and MCC are therefore mathematically not applicable, rather than unknown.
| Stemmer | Optimistic over-stemming (OI) | Optimistic under-stemming (UI) |
|---|---|---|
| Radixor | 0.000000% | 0.000000% |
Oracle-bound pair counts
| Stemmer | Unavoidable over errors / gold-negative pairs | Unrepairable under errors / gold-related pairs |
|---|---|---|
| Radixor | 0 / 288722133 | 0 / 106462 |
ALL_CANDIDATES result (no same-language comparator)
| Rank | Stemmer | Balanced accuracy | Over-stemming (OI) | Under-stemming (UI) |
|---|---|---|---|---|
| n/a | Radixor | 1.000000 | 0.000000% | 0.000000% |
Classification metrics
| Rank | Stemmer | Output policy | Precision | Recall | Specificity | Balanced accuracy | Pairwise accuracy | Error rate |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 0.000000 |
Pair-relation metrics
| Rank | Stemmer | Output policy | F0.5 | F1 | F2 | Jaccard | Fowlkes–Mallows | MCC |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 |
Raw pair counts
| Rank | Stemmer | Output policy | TP | FP | FN | TN | Over error / possible | Under error / possible |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 106462 | 0 | 0 | 288722133 | 0 / 288722133 | 0 / 106462 |
Multi-output analysis
Alternative candidates are capability analyses, not replacements for the deterministic comparison.
| Stemmer | Under pairs repaired | Best-case over pairs avoided | All-candidate collisions added | Multi-candidate forms | Multi-candidate share | Maximum candidates | Total candidate assignments |
|---|---|---|---|---|---|---|---|
| Radixor | 5179 | 0 | 0 | 454 | 1.888912% | 18 | 24533 |
LOWERCASE_GROUPS_ONLY
This mode contains 5 result rows, 3 evaluated stemmers, and 3 output policies. Applied-row and form counts are shown per row because adapters share the language corpus but policy rows remain independently auditable. Where at least two deterministic rows are available, PRIMARY_OUTPUT and ALL_CANDIDATES rankings are ordered by unrounded balanced accuracy, followed by MCC, F1, over-stemming rate, over-stemming count, under-stemming rate, and stemmer. ANY_CANDIDATE has no single rank metric and is listed alphabetically. Balanced accuracy is a navigation metric, not a universally authoritative quality score.
PRIMARY_OUTPUT ranking
| Rank | Stemmer | Balanced accuracy | Over-stemming (OI) | Under-stemming (UI) |
|---|---|---|---|---|
| 1 | Radixor | 0.975677 | 0.000000% | 4.864647% |
| 2 | SNOWBALL IRISH DIRECT | 0.512787 | 0.000290% | 97.442280% |
| 3 | SNOWBALL IRISH LUCENE FILTER | 0.512787 | 0.000290% | 97.442280% |
Classification metrics
| Rank | Stemmer | Output policy | Precision | Recall | Specificity | Balanced accuracy | Pairwise accuracy | Error rate |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 1.000000 | 0.951354 | 1.000000 | 0.975677 | 0.999982 | 0.000018 |
| 2 | SNOWBALL IRISH DIRECT | PRIMARY_OUTPUT | 0.764888 | 0.025577 | 0.999997 | 0.512787 | 0.999638 | 0.000362 |
| 3 | SNOWBALL IRISH LUCENE FILTER | PRIMARY_OUTPUT | 0.764888 | 0.025577 | 0.999997 | 0.512787 | 0.999638 | 0.000362 |
Pair-relation metrics
| Rank | Stemmer | Output policy | F0.5 | F1 | F2 | Jaccard | Fowlkes–Mallows | MCC |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 0.989877 | 0.975070 | 0.960700 | 0.951354 | 0.975374 | 0.975365 |
| 2 | SNOWBALL IRISH DIRECT | PRIMARY_OUTPUT | 0.112798 | 0.049499 | 0.031706 | 0.025378 | 0.139870 | 0.139829 |
| 3 | SNOWBALL IRISH LUCENE FILTER | PRIMARY_OUTPUT | 0.112798 | 0.049499 | 0.031706 | 0.025378 | 0.139870 | 0.139829 |
Raw pair counts
| Rank | Stemmer | Output policy | TP | FP | FN | TN | Over error / possible | Under error / possible |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 101283 | 0 | 5179 | 288722133 | 0 / 288722133 | 5179 / 106462 |
| 2 | SNOWBALL IRISH DIRECT | PRIMARY_OUTPUT | 2723 | 837 | 103739 | 288721296 | 837 / 288722133 | 103739 / 106462 |
| 3 | SNOWBALL IRISH LUCENE FILTER | PRIMARY_OUTPUT | 2723 | 837 | 103739 | 288721296 | 837 / 288722133 | 103739 / 106462 |
ANY_CANDIDATE oracle bounds
These results are measured, not missing. ANY_CANDIDATE answers two separate optimistic questions for each pair: a gold-related pair avoids under-stemming when the candidate sets intersect, while a gold-negative pair avoids over-stemming when some non-colliding candidate selection exists. The oracle may choose a different candidate for the same word in different pairs. Consequently, these decisions do not form one globally realizable predicted relation or one TP/FP/FN/TN confusion matrix. Balanced accuracy, F-scores, Jaccard, Fowlkes–Mallows, and MCC are therefore mathematically not applicable, rather than unknown.
| Stemmer | Optimistic over-stemming (OI) | Optimistic under-stemming (UI) |
|---|---|---|
| Radixor | 0.000000% | 0.000000% |
Oracle-bound pair counts
| Stemmer | Unavoidable over errors / gold-negative pairs | Unrepairable under errors / gold-related pairs |
|---|---|---|
| Radixor | 0 / 288722133 | 0 / 106462 |
ALL_CANDIDATES result (no same-language comparator)
| Rank | Stemmer | Balanced accuracy | Over-stemming (OI) | Under-stemming (UI) |
|---|---|---|---|---|
| n/a | Radixor | 1.000000 | 0.000000% | 0.000000% |
Classification metrics
| Rank | Stemmer | Output policy | Precision | Recall | Specificity | Balanced accuracy | Pairwise accuracy | Error rate |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 0.000000 |
Pair-relation metrics
| Rank | Stemmer | Output policy | F0.5 | F1 | F2 | Jaccard | Fowlkes–Mallows | MCC |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 |
Raw pair counts
| Rank | Stemmer | Output policy | TP | FP | FN | TN | Over error / possible | Under error / possible |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 106462 | 0 | 0 | 288722133 | 0 / 288722133 | 0 / 106462 |
Multi-output analysis
Alternative candidates are capability analyses, not replacements for the deterministic comparison.
| Stemmer | Under pairs repaired | Best-case over pairs avoided | All-candidate collisions added | Multi-candidate forms | Multi-candidate share | Maximum candidates | Total candidate assignments |
|---|---|---|---|---|---|---|---|
| Radixor | 5179 | 0 | 0 | 454 | 1.888912% | 18 | 24533 |
Method and interpretation boundaries
The linguistic-quality methodology defines the overlapping gold relation, both dictionary-processing modes, all three output policies, confusion counts, formulas, undefined denominators, aggregation, ranking, and inapplicable partition metrics. The candidate-policy reference explains why ANY_CANDIDATE is an oracle-assisted capability bound rather than deterministic runtime behavior.
This page preserves the language-specific raw counts, metrics, candidate distributions, comparison availability, and caveats. The machine-readable CSV remains authoritative for every field.
Provenance
- Authoritative source:
docs/benchmarks/data/stemming-quality-2026-09-13.csv - Source SHA-256:
d41e00160cda44758e806c37d210f2d9b90b1ebad22eebe5ece89a40f4ed9ab0 - Evaluation command:
./gradlew stemmingQuality --no-daemon - Dictionary language:
GA_IE - Processing modes:
ALL_WORDS,LOWERCASE_GROUPS_ONLY - Stemmer versions and transitive artifacts: resolved by the repository's JMH Gradle configuration and
gradle.lockfile - Model ID, version, and SHA-256: recorded in every CSV row
- Run date, core source state, JDK, operating system, and hardware: recorded on the benchmark environment page