Sotho, Southern stemmer benchmarks
This page reports dictionary corpus, exact-root agreement, and runtime evidence for the independently available st-za-default Sotho, Southern model.
Dictionary size: . The exact count is 416 distinct usable word forms after parser-compatible filtering and exact, case-preserved deduplication. Stars rank dictionary size relative to all benchmarked dictionaries in five nearly equal groups; they do not measure linguistic quality or benchmark accuracy.
The language metadata declares left-to-right writing.
All speed values are environment-specific and were measured on the hardware and JVM listed in the benchmark overview. The command distribution, exact-root accuracy, and speed tables belong to the published 2026-09-11 Radixor/Java 4.4.0 snapshot. Speed benchmark operations process changed tokens. Accuracy uses the complete Radixor dictionary for the language.
How to read this page
Start with the corpus and patch-command distribution, then compare exact-root agreement with runtime. Dictionary-size stars are contextual metadata, not an accuracy result.
Dictionary corpus
| Model ID | Model version | Language | Dictionary rows | Distinct usable forms | Complete quality tokens | Already-root tokens | Changed tokens | Timing workload | JMH timing tokens |
|---|---|---|---|---|---|---|---|---|---|
st-za-default |
1.0.0 |
ST_ZA |
26 | 416 | 416 | 26 | 390 | changed tokens | 5,000 |
Radixor patch-command distribution
Radixor stores the preferred transformation for each normalized dictionary word form as a compiled patch command. This distribution shows which runtime command class is selected by the trained trie for the complete default-model dictionary. The total number of preferred patch commands analyzed for this language is 416.
| Command class | Meaning | Word forms | Share |
|---|---|---|---|
BackwardCompoundCommand |
Applies a multi-step backward patch made from skip, delete, insert, and replace operations. | 364 | 87.500% |
DeleteSuffixCommand |
Deletes one or more trailing characters from the word form. | 26 | 6.250% |
PreserveCommand |
Returns the word form unchanged because it already matches the preferred root. | 26 | 6.250% |
Accuracy
Accuracy uses the complete dictionary and reports exact agreement with the dictionary root for each identified model and candidate.
| Stemmer | All exact | Changed exact | Root preserved | Note |
|---|---|---|---|---|
| Radixor | 100.000% | 100.000% | 100.000% | Exact model-ID benchmark; measured in this snapshot. |
| Official Snowball direct (Java) | 3.846% | 1.538% | 38.462% | Official Snowball 3.1.0 generated Java stemmer; measured in this snapshot. |
Speed
Speed uses JMH average time, 3 warmup iterations, 5 measurement iterations, 3 independent forks, and 1 thread.
The canonical timing workload uses changed tokens when available; a root-only dictionary uses its complete root-preservation corpus. Smaller populations are repeated deterministically to the timing minimum. Relative factors use the Radixor row as the baseline.
| Stemmer | Benchmark method | Score ms/op | Error ms | ns/token | Relative vs Radixor | Note |
|---|---|---|---|---|---|---|
| Radixor | radixor[st-za-default] |
0.250 | 0.064 | 50.1 | 1.000 | Canonical model timing workload; measured in this snapshot. |
| Official Snowball direct (Java) | snowballDirect[SESOTHO] |
0.266 | 0.062 | 53.1 | 1.061 | Official generated Java stemmer; measured in this snapshot. The reported Radixor and Snowball 99.9% JMH intervals overlap; this summary does not support a directional conclusion, and the runtime ratio remains a point estimate. |
Interpretation notes
- The star tier reflects only relative distinct-form count among all benchmarked dictionaries.
- Values shown above come from the identified canonical benchmark snapshot.
- Runtime and exact-root agreement describe different properties and must be interpreted together.
Dictionary-family generalization conclusion
This is the language-specific conclusion from the independent radixor-generalization-v1 baseline
experiment. It is intentionally separate from the wider edit-cost protocol below; values from
the two frozen snapshots are not substituted for one another.
Evidence
Model st-za-default version 1.0.0 is evaluated over five
predeclared nested splits. Unseen metrics remove withheld occurrences whose normalized surface
also appeared in training. Parentheses show the observed split minimum–maximum.
| Training rows | Median unseen occurrences | Unseen all exact | Unseen changed exact | Unseen root preserved |
|---|---|---|---|---|
| 10% | 368 | 12.500% | 6.667% | 100.000% |
| 20% | 336 | 12.500% | 6.667% | 100.000% |
| 30% | 288 | 12.500% | 6.667% | 100.000% |
| 40% | 256 | 12.500% | 6.667% | 100.000% |
| 50% | 208 | 12.500% | 6.667% | 100.000% |
| 60% | 160 | 12.500% | 6.667% | 100.000% |
| 70% | 128 | 12.500% | 6.667% | 100.000% |
| 80% | 80 | 12.500% | 6.667% | 100.000% |
| 90% | 48 | 12.500% | 6.667% | 100.000% |
Generalization conclusion
- Median exactness on genuinely unseen changed forms moves from 6.667% at 10% training knowledge to 6.667% at 90%, a measured +0.000 percentage-point change.
- Unseen all-form exactness moves from 12.500% at 10% training knowledge to 12.500% at 90%, a measured +0.000 percentage-point change.
- Preservation of unseen already-root forms moves from 100.000% at 10% training knowledge to 100.000% at 90%, a measured +0.000 percentage-point change.
- The evidence establishes within-resource transfer across withheld dictionary families. It does not estimate unrelated domains, misspellings, arbitrary compounds, or external corpora.
The complete ten-level table and split ranges remain in the complete generalization appendix; raw counters and provenance are in active machine-readable snapshot. The frozen methodology defines family-level splitting, unseen-surface leakage control, aggregation, and the limits of the claim.
Stemming quality
Runtime performance and linguistic grouping quality are independent dimensions. This section evaluates language ST_ZA using the complete validated stemming-quality result matrix and the canonical linguistic-quality methodology, including its overlapping-group relation, dictionary modes, output policies, metrics, and ranking rules.
Radixor's model was trained from the same lexical resource that defines this benchmark's dictionary-reference relation. The result therefore measures same-resource agreement, not independent external linguistic validity. Third-party adapters receive the same evaluated forms but were not trained by this benchmark.
The tables retain both canonical dictionary-processing modes and every applicable output policy without redefining them on each language page. Download the complete machine-readable result snapshot.
Evaluation scope and key findings
The default model is st-za-default, loaded from classpath resource org/egothor/stemmer/models/st-za-default/stemmer.gz. The following findings compare only deterministic PRIMARY_OUTPUT rows over identical included groups; candidate policies are reported separately as capability analyses.
- ALL_WORDS:
Radixorranks first by balanced accuracy at 1.000000 among 2 deterministic stemmers. The runner-up isSNOWBALL SESOTHO DIRECTat 0.503417, a difference of 0.496583. This rank does not imply leadership in throughput or every secondary metric. - LOWERCASE_GROUPS_ONLY:
Radixorranks first by balanced accuracy at 1.000000 among 2 deterministic stemmers. The runner-up isSNOWBALL SESOTHO DIRECTat 0.503417, a difference of 0.496583. This rank does not imply leadership in throughput or every secondary metric.
ALL_WORDS
This mode contains 4 result rows, 2 evaluated stemmers, and 3 output policies. Applied-row and form counts are shown per row because adapters share the language corpus but policy rows remain independently auditable. Where at least two deterministic rows are available, PRIMARY_OUTPUT and ALL_CANDIDATES rankings are ordered by unrounded balanced accuracy, followed by MCC, F1, over-stemming rate, over-stemming count, under-stemming rate, and stemmer. ANY_CANDIDATE has no single rank metric and is listed alphabetically. Balanced accuracy is a navigation metric, not a universally authoritative quality score.
PRIMARY_OUTPUT ranking
| Rank | Stemmer | Balanced accuracy | Over-stemming (OI) | Under-stemming (UI) |
|---|---|---|---|---|
| 1 | Radixor | 1.000000 | 0.000000% | 0.000000% |
| 2 | SNOWBALL SESOTHO DIRECT | 0.503417 | 0.021635% | 99.294872% |
Classification metrics
| Rank | Stemmer | Output policy | Precision | Recall | Specificity | Balanced accuracy | Pairwise accuracy | Error rate |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 0.000000 |
| 2 | SNOWBALL SESOTHO DIRECT | PRIMARY_OUTPUT | 0.550000 | 0.007051 | 0.999784 | 0.503417 | 0.963902 | 0.036098 |
Pair-relation metrics
| Rank | Stemmer | Output policy | F0.5 | F1 | F2 | Jaccard | Fowlkes–Mallows | MCC |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 |
| 2 | SNOWBALL SESOTHO DIRECT | PRIMARY_OUTPUT | 0.033537 | 0.013924 | 0.008786 | 0.007011 | 0.062275 | 0.059277 |
Raw pair counts
| Rank | Stemmer | Output policy | TP | FP | FN | TN | Over error / possible | Under error / possible |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 3120 | 0 | 0 | 83200 | 0 / 83200 | 0 / 3120 |
| 2 | SNOWBALL SESOTHO DIRECT | PRIMARY_OUTPUT | 22 | 18 | 3098 | 83182 | 18 / 83200 | 3098 / 3120 |
ANY_CANDIDATE oracle bounds
These results are measured, not missing. ANY_CANDIDATE answers two separate optimistic questions for each pair: a gold-related pair avoids under-stemming when the candidate sets intersect, while a gold-negative pair avoids over-stemming when some non-colliding candidate selection exists. The oracle may choose a different candidate for the same word in different pairs. Consequently, these decisions do not form one globally realizable predicted relation or one TP/FP/FN/TN confusion matrix. Balanced accuracy, F-scores, Jaccard, Fowlkes–Mallows, and MCC are therefore mathematically not applicable, rather than unknown.
| Stemmer | Optimistic over-stemming (OI) | Optimistic under-stemming (UI) |
|---|---|---|
| Radixor | 0.000000% | 0.000000% |
Oracle-bound pair counts
| Stemmer | Unavoidable over errors / gold-negative pairs | Unrepairable under errors / gold-related pairs |
|---|---|---|
| Radixor | 0 / 83200 | 0 / 3120 |
ALL_CANDIDATES result (no same-language comparator)
| Rank | Stemmer | Balanced accuracy | Over-stemming (OI) | Under-stemming (UI) |
|---|---|---|---|---|
| n/a | Radixor | 1.000000 | 0.000000% | 0.000000% |
Classification metrics
| Rank | Stemmer | Output policy | Precision | Recall | Specificity | Balanced accuracy | Pairwise accuracy | Error rate |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 0.000000 |
Pair-relation metrics
| Rank | Stemmer | Output policy | F0.5 | F1 | F2 | Jaccard | Fowlkes–Mallows | MCC |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 |
Raw pair counts
| Rank | Stemmer | Output policy | TP | FP | FN | TN | Over error / possible | Under error / possible |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 3120 | 0 | 0 | 83200 | 0 / 83200 | 0 / 3120 |
Multi-output analysis
Alternative candidates are capability analyses, not replacements for the deterministic comparison.
| Stemmer | Under pairs repaired | Best-case over pairs avoided | All-candidate collisions added | Multi-candidate forms | Multi-candidate share | Maximum candidates | Total candidate assignments |
|---|---|---|---|---|---|---|---|
| Radixor | 0 | 0 | 0 | 0 | 0.000000% | 1 | 416 |
LOWERCASE_GROUPS_ONLY
This mode contains 4 result rows, 2 evaluated stemmers, and 3 output policies. Applied-row and form counts are shown per row because adapters share the language corpus but policy rows remain independently auditable. Where at least two deterministic rows are available, PRIMARY_OUTPUT and ALL_CANDIDATES rankings are ordered by unrounded balanced accuracy, followed by MCC, F1, over-stemming rate, over-stemming count, under-stemming rate, and stemmer. ANY_CANDIDATE has no single rank metric and is listed alphabetically. Balanced accuracy is a navigation metric, not a universally authoritative quality score.
PRIMARY_OUTPUT ranking
| Rank | Stemmer | Balanced accuracy | Over-stemming (OI) | Under-stemming (UI) |
|---|---|---|---|---|
| 1 | Radixor | 1.000000 | 0.000000% | 0.000000% |
| 2 | SNOWBALL SESOTHO DIRECT | 0.503417 | 0.021635% | 99.294872% |
Classification metrics
| Rank | Stemmer | Output policy | Precision | Recall | Specificity | Balanced accuracy | Pairwise accuracy | Error rate |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 0.000000 |
| 2 | SNOWBALL SESOTHO DIRECT | PRIMARY_OUTPUT | 0.550000 | 0.007051 | 0.999784 | 0.503417 | 0.963902 | 0.036098 |
Pair-relation metrics
| Rank | Stemmer | Output policy | F0.5 | F1 | F2 | Jaccard | Fowlkes–Mallows | MCC |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 |
| 2 | SNOWBALL SESOTHO DIRECT | PRIMARY_OUTPUT | 0.033537 | 0.013924 | 0.008786 | 0.007011 | 0.062275 | 0.059277 |
Raw pair counts
| Rank | Stemmer | Output policy | TP | FP | FN | TN | Over error / possible | Under error / possible |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 3120 | 0 | 0 | 83200 | 0 / 83200 | 0 / 3120 |
| 2 | SNOWBALL SESOTHO DIRECT | PRIMARY_OUTPUT | 22 | 18 | 3098 | 83182 | 18 / 83200 | 3098 / 3120 |
ANY_CANDIDATE oracle bounds
These results are measured, not missing. ANY_CANDIDATE answers two separate optimistic questions for each pair: a gold-related pair avoids under-stemming when the candidate sets intersect, while a gold-negative pair avoids over-stemming when some non-colliding candidate selection exists. The oracle may choose a different candidate for the same word in different pairs. Consequently, these decisions do not form one globally realizable predicted relation or one TP/FP/FN/TN confusion matrix. Balanced accuracy, F-scores, Jaccard, Fowlkes–Mallows, and MCC are therefore mathematically not applicable, rather than unknown.
| Stemmer | Optimistic over-stemming (OI) | Optimistic under-stemming (UI) |
|---|---|---|
| Radixor | 0.000000% | 0.000000% |
Oracle-bound pair counts
| Stemmer | Unavoidable over errors / gold-negative pairs | Unrepairable under errors / gold-related pairs |
|---|---|---|
| Radixor | 0 / 83200 | 0 / 3120 |
ALL_CANDIDATES result (no same-language comparator)
| Rank | Stemmer | Balanced accuracy | Over-stemming (OI) | Under-stemming (UI) |
|---|---|---|---|---|
| n/a | Radixor | 1.000000 | 0.000000% | 0.000000% |
Classification metrics
| Rank | Stemmer | Output policy | Precision | Recall | Specificity | Balanced accuracy | Pairwise accuracy | Error rate |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 0.000000 |
Pair-relation metrics
| Rank | Stemmer | Output policy | F0.5 | F1 | F2 | Jaccard | Fowlkes–Mallows | MCC |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 |
Raw pair counts
| Rank | Stemmer | Output policy | TP | FP | FN | TN | Over error / possible | Under error / possible |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 3120 | 0 | 0 | 83200 | 0 / 83200 | 0 / 3120 |
Multi-output analysis
Alternative candidates are capability analyses, not replacements for the deterministic comparison.
| Stemmer | Under pairs repaired | Best-case over pairs avoided | All-candidate collisions added | Multi-candidate forms | Multi-candidate share | Maximum candidates | Total candidate assignments |
|---|---|---|---|---|---|---|---|
| Radixor | 0 | 0 | 0 | 0 | 0.000000% | 1 | 416 |
Method and interpretation boundaries
The linguistic-quality methodology defines the overlapping gold relation, both dictionary-processing modes, all three output policies, confusion counts, formulas, undefined denominators, aggregation, ranking, and inapplicable partition metrics. The candidate-policy reference explains why ANY_CANDIDATE is an oracle-assisted capability bound rather than deterministic runtime behavior.
This page preserves the language-specific raw counts, metrics, candidate distributions, comparison availability, and caveats. The machine-readable CSV remains authoritative for every field.
Provenance
- Authoritative source:
docs/benchmarks/data/stemming-quality-2026-09-13.csv - Source SHA-256:
d41e00160cda44758e806c37d210f2d9b90b1ebad22eebe5ece89a40f4ed9ab0 - Evaluation command:
./gradlew stemmingQuality --no-daemon - Dictionary language:
ST_ZA - Processing modes:
ALL_WORDS,LOWERCASE_GROUPS_ONLY - Stemmer versions and transitive artifacts: resolved by the repository's JMH Gradle configuration and
gradle.lockfile - Model ID, version, and SHA-256: recorded in every CSV row
- Run date, core source state, JDK, operating system, and hardware: recorded on the benchmark environment page