Armenian stemmer benchmarks
This page reports dictionary corpus, exact-root agreement, and runtime evidence for the independently available hy-am-default Armenian model.
Dictionary size: . The exact count is 246,576 distinct usable word forms after parser-compatible filtering and exact, case-preserved deduplication. Stars rank dictionary size relative to all benchmarked dictionaries in five nearly equal groups; they do not measure linguistic quality or benchmark accuracy.
The language metadata declares left-to-right writing.
All speed values are environment-specific and were measured on the hardware and JVM listed in the benchmark overview. The command distribution, exact-root accuracy, and speed tables belong to the published 2026-09-11 Radixor/Java 4.4.0 snapshot. Speed benchmark operations process changed tokens. Accuracy uses the complete Radixor dictionary for the language.
How to read this page
Start with the corpus and patch-command distribution, then compare exact-root agreement with runtime. Dictionary-size stars are contextual metadata, not an accuracy result.
Dictionary corpus
| Model ID | Model version | Language | Dictionary rows | Distinct usable forms | Complete quality tokens | Already-root tokens | Changed tokens | Timing workload | JMH timing tokens |
|---|---|---|---|---|---|---|---|---|---|
hy-am-default |
1.0.0 |
HY_AM |
6,990 | 246,576 | 247,803 | 6,990 | 240,813 | changed tokens | 240,813 |
Radixor patch-command distribution
Radixor stores the preferred transformation for each normalized dictionary word form as a compiled patch command. This distribution shows which runtime command class is selected by the trained trie for the complete default-model dictionary. The total number of preferred patch commands analyzed for this language is 247,803.
| Command class | Meaning | Word forms | Share |
|---|---|---|---|
AppendCharacterCommand |
Appends one character to the end of the word form. | 161 | 0.065% |
BackwardCompoundCommand |
Applies a multi-step backward patch made from skip, delete, insert, and replace operations. | 56,059 | 22.622% |
DeleteSuffixCommand |
Deletes one or more trailing characters from the word form. | 178,362 | 71.977% |
PreserveCommand |
Returns the word form unchanged because it already matches the preferred root. | 7,207 | 2.908% |
ReplaceLastCharacterCommand |
Replaces the final character of the word form. | 6,014 | 2.427% |
Accuracy
Accuracy uses the complete dictionary and reports exact agreement with the dictionary root for each identified model and candidate.
| Stemmer | All exact | Changed exact | Root preserved | Note |
|---|---|---|---|---|
| Radixor | 99.505% | 99.525% | 98.827% | Exact model-ID benchmark; measured in this snapshot. |
| Official Snowball direct (Java) | 12.656% | 11.867% | 39.814% | Official Snowball 3.1.0 generated Java stemmer; measured in this snapshot. |
| Lucene SnowballFilter | 12.656% | 11.867% | 39.814% | Lucene integration of the matching Snowball algorithm; measured in this snapshot. |
Speed
Speed uses JMH average time, 3 warmup iterations, 5 measurement iterations, 3 independent forks, and 1 thread.
The canonical timing workload uses changed tokens when available; a root-only dictionary uses its complete root-preservation corpus. Smaller populations are repeated deterministically to the timing minimum. Relative factors use the Radixor row as the baseline.
| Stemmer | Benchmark method | Score ms/op | Error ms | ns/token | Relative vs Radixor | Note |
|---|---|---|---|---|---|---|
| Radixor | radixor[hy-am-default] |
21.413 | 4.712 | 88.9 | 1.000 | Canonical model timing workload; measured in this snapshot. |
| Official Snowball direct (Java) | snowballDirect[ARMENIAN] |
35.200 | 4.623 | 146.2 | 1.644 | Official generated Java stemmer; measured in this snapshot. |
| Lucene SnowballFilter | luceneSnowballFilter[ARMENIAN] |
50.006 | 5.424 | 207.7 | 2.335 | Lucene TokenStream integration; measured in this snapshot. |
Interpretation notes
- The star tier reflects only relative distinct-form count among all benchmarked dictionaries.
- Values shown above come from the identified canonical benchmark snapshot.
- Runtime and exact-root agreement describe different properties and must be interpreted together.
Dictionary-family generalization conclusion
This is the language-specific conclusion from the independent radixor-generalization-v1 baseline
experiment. It is intentionally separate from the wider edit-cost protocol below; values from
the two frozen snapshots are not substituted for one another.
Evidence
Model hy-am-default version 1.0.0 is evaluated over five
predeclared nested splits. Unseen metrics remove withheld occurrences whose normalized surface
also appeared in training. Parentheses show the observed split minimum–maximum.
| Training rows | Median unseen occurrences | Unseen all exact | Unseen changed exact | Unseen root preserved |
|---|---|---|---|---|
| 10% | 223,028 | 65.016% (64.402–66.160) | 64.272% (63.678–65.429) | 90.256% (89.494–91.529) |
| 20% | 198,108 | 66.540% (65.927–68.287) | 65.815% (65.216–67.626) | 91.304% (90.282–91.762) |
| 30% | 173,169 | 67.599% (66.815–68.266) | 66.900% (66.110–67.588) | 91.549% (90.862–92.046) |
| 40% | 148,341 | 68.280% (67.572–68.505) | 67.598% (66.889–67.827) | 92.302% (91.640–92.490) |
| 50% | 123,633 | 68.911% (68.483–69.122) | 68.226% (67.807–68.446) | 92.769% (92.473–93.117) |
| 60% | 98,745 | 69.346% (68.769–70.333) | 68.695% (68.106–69.680) | 93.306% (92.451–93.456) |
| 70% | 74,063 | 69.445% (69.174–70.424) | 68.763% (68.515–69.745) | 93.373% (92.537–94.431) |
| 80% | 49,373 | 69.925% (68.793–71.043) | 69.267% (68.105–70.382) | 93.568% (93.328–94.654) |
| 90% | 24,487 | 69.740% (68.616–72.259) | 69.080% (67.922–71.648) | 93.835% (93.383–95.152) |
Generalization conclusion
- Median exactness on genuinely unseen changed forms moves from 64.272% at 10% training knowledge to 69.080% at 90%, a measured +4.808 percentage-point change.
- Unseen all-form exactness moves from 65.016% at 10% training knowledge to 69.740% at 90%, a measured +4.724 percentage-point change.
- Preservation of unseen already-root forms moves from 90.256% at 10% training knowledge to 93.835% at 90%, a measured +3.579 percentage-point change.
- The evidence establishes within-resource transfer across withheld dictionary families. It does not estimate unrelated domains, misspellings, arbitrary compounds, or external corpora.
The complete ten-level table and split ranges remain in the complete generalization appendix; raw counters and provenance are in active machine-readable snapshot. The frozen methodology defines family-level splitting, unseen-surface leakage control, aggregation, and the limits of the claim.
Stemming quality
Runtime performance and linguistic grouping quality are independent dimensions. This section evaluates language HY_AM using the complete validated stemming-quality result matrix and the canonical linguistic-quality methodology, including its overlapping-group relation, dictionary modes, output policies, metrics, and ranking rules.
Radixor's model was trained from the same lexical resource that defines this benchmark's dictionary-reference relation. The result therefore measures same-resource agreement, not independent external linguistic validity. Third-party adapters receive the same evaluated forms but were not trained by this benchmark.
The tables retain both canonical dictionary-processing modes and every applicable output policy without redefining them on each language page. Download the complete machine-readable result snapshot.
Evaluation scope and key findings
The default model is hy-am-default, loaded from classpath resource org/egothor/stemmer/models/hy-am-default/stemmer.gz. The following findings compare only deterministic PRIMARY_OUTPUT rows over identical included groups; candidate policies are reported separately as capability analyses.
- ALL_WORDS:
Radixorranks first by balanced accuracy at 0.993984 among 3 deterministic stemmers. The runner-up isSNOWBALL ARMENIAN DIRECTat 0.566158, a difference of 0.427826. This rank does not imply leadership in throughput or every secondary metric. - LOWERCASE_GROUPS_ONLY:
Radixorranks first by balanced accuracy at 0.993984 among 3 deterministic stemmers. The runner-up isSNOWBALL ARMENIAN DIRECTat 0.566158, a difference of 0.427826. This rank does not imply leadership in throughput or every secondary metric.
ALL_WORDS
This mode contains 5 result rows, 3 evaluated stemmers, and 3 output policies. Applied-row and form counts are shown per row because adapters share the language corpus but policy rows remain independently auditable. Where at least two deterministic rows are available, PRIMARY_OUTPUT and ALL_CANDIDATES rankings are ordered by unrounded balanced accuracy, followed by MCC, F1, over-stemming rate, over-stemming count, under-stemming rate, and stemmer. ANY_CANDIDATE has no single rank metric and is listed alphabetically. Balanced accuracy is a navigation metric, not a universally authoritative quality score.
PRIMARY_OUTPUT ranking
| Rank | Stemmer | Balanced accuracy | Over-stemming (OI) | Under-stemming (UI) |
|---|---|---|---|---|
| 1 | Radixor | 0.993984 | 0.000000% | 1.203238% |
| 2 | SNOWBALL ARMENIAN DIRECT | 0.566158 | 0.001021% | 86.767379% |
| 3 | SNOWBALL ARMENIAN LUCENE FILTER | 0.566158 | 0.001021% | 86.767379% |
Classification metrics
| Rank | Stemmer | Output policy | Precision | Recall | Specificity | Balanced accuracy | Pairwise accuracy | Error rate |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 1.000000 | 0.987968 | 1.000000 | 0.993984 | 0.999998 | 0.000002 |
| 2 | SNOWBALL ARMENIAN DIRECT | PRIMARY_OUTPUT | 0.667460 | 0.132326 | 0.999990 | 0.566158 | 0.999855 | 0.000145 |
| 3 | SNOWBALL ARMENIAN LUCENE FILTER | PRIMARY_OUTPUT | 0.667460 | 0.132326 | 0.999990 | 0.566158 | 0.999855 | 0.000145 |
Pair-relation metrics
| Rank | Stemmer | Output policy | F0.5 | F1 | F2 | Jaccard | Fowlkes–Mallows | MCC |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 0.997570 | 0.993947 | 0.990351 | 0.987968 | 0.993966 | 0.993965 |
| 2 | SNOWBALL ARMENIAN DIRECT | PRIMARY_OUTPUT | 0.369005 | 0.220865 | 0.157597 | 0.124142 | 0.297191 | 0.297149 |
| 3 | SNOWBALL ARMENIAN LUCENE FILTER | PRIMARY_OUTPUT | 0.369005 | 0.220865 | 0.157597 | 0.124142 | 0.297191 | 0.297149 |
Raw pair counts
| Rank | Stemmer | Output policy | TP | FP | FN | TN | Over error / possible | Under error / possible |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 4650084 | 0 | 56633 | 30395031883 | 0 / 30395031883 | 56633 / 4706717 |
| 2 | SNOWBALL ARMENIAN DIRECT | PRIMARY_OUTPUT | 622822 | 310301 | 4083895 | 30394721582 | 310301 / 30395031883 | 4083895 / 4706717 |
| 3 | SNOWBALL ARMENIAN LUCENE FILTER | PRIMARY_OUTPUT | 622822 | 310301 | 4083895 | 30394721582 | 310301 / 30395031883 | 4083895 / 4706717 |
ANY_CANDIDATE oracle bounds
These results are measured, not missing. ANY_CANDIDATE answers two separate optimistic questions for each pair: a gold-related pair avoids under-stemming when the candidate sets intersect, while a gold-negative pair avoids over-stemming when some non-colliding candidate selection exists. The oracle may choose a different candidate for the same word in different pairs. Consequently, these decisions do not form one globally realizable predicted relation or one TP/FP/FN/TN confusion matrix. Balanced accuracy, F-scores, Jaccard, Fowlkes–Mallows, and MCC are therefore mathematically not applicable, rather than unknown.
| Stemmer | Optimistic over-stemming (OI) | Optimistic under-stemming (UI) |
|---|---|---|
| Radixor | 0.000000% | 0.000000% |
Oracle-bound pair counts
| Stemmer | Unavoidable over errors / gold-negative pairs | Unrepairable under errors / gold-related pairs |
|---|---|---|
| Radixor | 0 / 30395031883 | 0 / 4706717 |
ALL_CANDIDATES result (no same-language comparator)
| Rank | Stemmer | Balanced accuracy | Over-stemming (OI) | Under-stemming (UI) |
|---|---|---|---|---|
| n/a | Radixor | 1.000000 | 0.000000% | 0.000000% |
Classification metrics
| Rank | Stemmer | Output policy | Precision | Recall | Specificity | Balanced accuracy | Pairwise accuracy | Error rate |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 0.000000 |
Pair-relation metrics
| Rank | Stemmer | Output policy | F0.5 | F1 | F2 | Jaccard | Fowlkes–Mallows | MCC |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 |
Raw pair counts
| Rank | Stemmer | Output policy | TP | FP | FN | TN | Over error / possible | Under error / possible |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 4706717 | 0 | 0 | 30395031883 | 0 / 30395031883 | 0 / 4706717 |
Multi-output analysis
Alternative candidates are capability analyses, not replacements for the deterministic comparison.
| Stemmer | Under pairs repaired | Best-case over pairs avoided | All-candidate collisions added | Multi-candidate forms | Multi-candidate share | Maximum candidates | Total candidate assignments |
|---|---|---|---|---|---|---|---|
| Radixor | 56633 | 0 | 0 | 1200 | 0.486665% | 4 | 247803 |
LOWERCASE_GROUPS_ONLY
This mode contains 5 result rows, 3 evaluated stemmers, and 3 output policies. Applied-row and form counts are shown per row because adapters share the language corpus but policy rows remain independently auditable. Where at least two deterministic rows are available, PRIMARY_OUTPUT and ALL_CANDIDATES rankings are ordered by unrounded balanced accuracy, followed by MCC, F1, over-stemming rate, over-stemming count, under-stemming rate, and stemmer. ANY_CANDIDATE has no single rank metric and is listed alphabetically. Balanced accuracy is a navigation metric, not a universally authoritative quality score.
PRIMARY_OUTPUT ranking
| Rank | Stemmer | Balanced accuracy | Over-stemming (OI) | Under-stemming (UI) |
|---|---|---|---|---|
| 1 | Radixor | 0.993984 | 0.000000% | 1.203238% |
| 2 | SNOWBALL ARMENIAN DIRECT | 0.566158 | 0.001021% | 86.767379% |
| 3 | SNOWBALL ARMENIAN LUCENE FILTER | 0.566158 | 0.001021% | 86.767379% |
Classification metrics
| Rank | Stemmer | Output policy | Precision | Recall | Specificity | Balanced accuracy | Pairwise accuracy | Error rate |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 1.000000 | 0.987968 | 1.000000 | 0.993984 | 0.999998 | 0.000002 |
| 2 | SNOWBALL ARMENIAN DIRECT | PRIMARY_OUTPUT | 0.667460 | 0.132326 | 0.999990 | 0.566158 | 0.999855 | 0.000145 |
| 3 | SNOWBALL ARMENIAN LUCENE FILTER | PRIMARY_OUTPUT | 0.667460 | 0.132326 | 0.999990 | 0.566158 | 0.999855 | 0.000145 |
Pair-relation metrics
| Rank | Stemmer | Output policy | F0.5 | F1 | F2 | Jaccard | Fowlkes–Mallows | MCC |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 0.997570 | 0.993947 | 0.990351 | 0.987968 | 0.993966 | 0.993965 |
| 2 | SNOWBALL ARMENIAN DIRECT | PRIMARY_OUTPUT | 0.369005 | 0.220865 | 0.157597 | 0.124142 | 0.297191 | 0.297149 |
| 3 | SNOWBALL ARMENIAN LUCENE FILTER | PRIMARY_OUTPUT | 0.369005 | 0.220865 | 0.157597 | 0.124142 | 0.297191 | 0.297149 |
Raw pair counts
| Rank | Stemmer | Output policy | TP | FP | FN | TN | Over error / possible | Under error / possible |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 4650084 | 0 | 56633 | 30395031883 | 0 / 30395031883 | 56633 / 4706717 |
| 2 | SNOWBALL ARMENIAN DIRECT | PRIMARY_OUTPUT | 622822 | 310301 | 4083895 | 30394721582 | 310301 / 30395031883 | 4083895 / 4706717 |
| 3 | SNOWBALL ARMENIAN LUCENE FILTER | PRIMARY_OUTPUT | 622822 | 310301 | 4083895 | 30394721582 | 310301 / 30395031883 | 4083895 / 4706717 |
ANY_CANDIDATE oracle bounds
These results are measured, not missing. ANY_CANDIDATE answers two separate optimistic questions for each pair: a gold-related pair avoids under-stemming when the candidate sets intersect, while a gold-negative pair avoids over-stemming when some non-colliding candidate selection exists. The oracle may choose a different candidate for the same word in different pairs. Consequently, these decisions do not form one globally realizable predicted relation or one TP/FP/FN/TN confusion matrix. Balanced accuracy, F-scores, Jaccard, Fowlkes–Mallows, and MCC are therefore mathematically not applicable, rather than unknown.
| Stemmer | Optimistic over-stemming (OI) | Optimistic under-stemming (UI) |
|---|---|---|
| Radixor | 0.000000% | 0.000000% |
Oracle-bound pair counts
| Stemmer | Unavoidable over errors / gold-negative pairs | Unrepairable under errors / gold-related pairs |
|---|---|---|
| Radixor | 0 / 30395031883 | 0 / 4706717 |
ALL_CANDIDATES result (no same-language comparator)
| Rank | Stemmer | Balanced accuracy | Over-stemming (OI) | Under-stemming (UI) |
|---|---|---|---|---|
| n/a | Radixor | 1.000000 | 0.000000% | 0.000000% |
Classification metrics
| Rank | Stemmer | Output policy | Precision | Recall | Specificity | Balanced accuracy | Pairwise accuracy | Error rate |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 0.000000 |
Pair-relation metrics
| Rank | Stemmer | Output policy | F0.5 | F1 | F2 | Jaccard | Fowlkes–Mallows | MCC |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 |
Raw pair counts
| Rank | Stemmer | Output policy | TP | FP | FN | TN | Over error / possible | Under error / possible |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 4706717 | 0 | 0 | 30395031883 | 0 / 30395031883 | 0 / 4706717 |
Multi-output analysis
Alternative candidates are capability analyses, not replacements for the deterministic comparison.
| Stemmer | Under pairs repaired | Best-case over pairs avoided | All-candidate collisions added | Multi-candidate forms | Multi-candidate share | Maximum candidates | Total candidate assignments |
|---|---|---|---|---|---|---|---|
| Radixor | 56633 | 0 | 0 | 1200 | 0.486665% | 4 | 247803 |
Method and interpretation boundaries
The linguistic-quality methodology defines the overlapping gold relation, both dictionary-processing modes, all three output policies, confusion counts, formulas, undefined denominators, aggregation, ranking, and inapplicable partition metrics. The candidate-policy reference explains why ANY_CANDIDATE is an oracle-assisted capability bound rather than deterministic runtime behavior.
This page preserves the language-specific raw counts, metrics, candidate distributions, comparison availability, and caveats. The machine-readable CSV remains authoritative for every field.
Provenance
- Authoritative source:
docs/benchmarks/data/stemming-quality-2026-09-13.csv - Source SHA-256:
d41e00160cda44758e806c37d210f2d9b90b1ebad22eebe5ece89a40f4ed9ab0 - Evaluation command:
./gradlew stemmingQuality --no-daemon - Dictionary language:
HY_AM - Processing modes:
ALL_WORDS,LOWERCASE_GROUPS_ONLY - Stemmer versions and transitive artifacts: resolved by the repository's JMH Gradle configuration and
gradle.lockfile - Model ID, version, and SHA-256: recorded in every CSV row
- Run date, core source state, JDK, operating system, and hardware: recorded on the benchmark environment page