Turkish stemmer benchmarks
This page reports dictionary corpus, exact-root agreement, and runtime evidence for the independently available tr-tr-default Turkish model.
Dictionary size: . The exact count is 222,553 distinct usable word forms after parser-compatible filtering and exact, case-preserved deduplication. Stars rank dictionary size relative to all benchmarked dictionaries in five nearly equal groups; they do not measure linguistic quality or benchmark accuracy.
The language metadata declares left-to-right writing.
All speed values are environment-specific and were measured on the hardware and JVM listed in the benchmark overview. The command distribution, exact-root accuracy, and speed tables belong to the published 2026-09-11 Radixor/Java 4.4.0 snapshot. Speed benchmark operations process changed tokens. Accuracy uses the complete Radixor dictionary for the language.
How to read this page
Start with the corpus and patch-command distribution, then compare exact-root agreement with runtime. Dictionary-size stars are contextual metadata, not an accuracy result.
Dictionary corpus
| Model ID | Model version | Language | Dictionary rows | Distinct usable forms | Complete quality tokens | Already-root tokens | Changed tokens | Timing workload | JMH timing tokens |
|---|---|---|---|---|---|---|---|---|---|
tr-tr-default |
1.0.0 |
TR_TR |
3,017 | 222,553 | 224,205 | 3,017 | 221,188 | changed tokens | 221,188 |
Radixor patch-command distribution
Radixor stores the preferred transformation for each normalized dictionary word form as a compiled patch command. This distribution shows which runtime command class is selected by the trained trie for the complete default-model dictionary. The total number of preferred patch commands analyzed for this language is 224,205.
| Command class | Meaning | Word forms | Share |
|---|---|---|---|
AppendCharacterCommand |
Appends one character to the end of the word form. | 568 | 0.253% |
BackwardCompoundCommand |
Applies a multi-step backward patch made from skip, delete, insert, and replace operations. | 116,469 | 51.948% |
DeleteSuffixCommand |
Deletes one or more trailing characters from the word form. | 103,111 | 45.990% |
PreserveCommand |
Returns the word form unchanged because it already matches the preferred root. | 3,005 | 1.340% |
ReplaceLastCharacterCommand |
Replaces the final character of the word form. | 1,052 | 0.469% |
Accuracy
Accuracy uses the complete dictionary and reports exact agreement with the dictionary root for each identified model and candidate.
| Stemmer | All exact | Changed exact | Root preserved | Note |
|---|---|---|---|---|
| Radixor | 99.263% | 99.280% | 98.044% | Exact model-ID benchmark; measured in this snapshot. |
| Official Snowball direct (Java) | 44.149% | 43.751% | 73.351% | Official Snowball 3.1.0 generated Java stemmer; measured in this snapshot. |
| Lucene SnowballFilter | 44.133% | 43.734% | 73.351% | Lucene integration of the matching Snowball algorithm; measured in this snapshot. |
Speed
Speed uses JMH average time, 3 warmup iterations, 5 measurement iterations, 3 independent forks, and 1 thread.
The canonical timing workload uses changed tokens when available; a root-only dictionary uses its complete root-preservation corpus. Smaller populations are repeated deterministically to the timing minimum. Relative factors use the Radixor row as the baseline.
| Stemmer | Benchmark method | Score ms/op | Error ms | ns/token | Relative vs Radixor | Note |
|---|---|---|---|---|---|---|
| Radixor | radixor[tr-tr-default] |
25.962 | 2.892 | 117.4 | 1.000 | Canonical model timing workload; measured in this snapshot. |
| Official Snowball direct (Java) | snowballDirect[TURKISH] |
97.758 | 4.259 | 442.0 | 3.765 | Official generated Java stemmer; measured in this snapshot. |
| Lucene SnowballFilter | luceneSnowballFilter[TURKISH] |
102.162 | 3.781 | 461.9 | 3.935 | Lucene TokenStream integration; measured in this snapshot. |
Interpretation notes
- The star tier reflects only relative distinct-form count among all benchmarked dictionaries.
- Values shown above come from the identified canonical benchmark snapshot.
- Runtime and exact-root agreement describe different properties and must be interpreted together.
Dictionary-family generalization conclusion
This is the language-specific conclusion from the independent radixor-generalization-v1 baseline
experiment. It is intentionally separate from the wider edit-cost protocol below; values from
the two frozen snapshots are not substituted for one another.
Evidence
Model tr-tr-default version 1.0.0 is evaluated over five
predeclared nested splits. Unseen metrics remove withheld occurrences whose normalized surface
also appeared in training. Parentheses show the observed split minimum–maximum.
| Training rows | Median unseen occurrences | Unseen all exact | Unseen changed exact | Unseen root preserved |
|---|---|---|---|---|
| 10% | 201,851 | 77.397% (74.873–79.731) | 77.296% (74.748–79.658) | 84.869% (83.617–85.031) |
| 20% | 178,334 | 79.471% (77.942–80.778) | 79.417% (77.814–80.700) | 86.502% (83.375–87.385) |
| 30% | 155,798 | 81.433% (79.698–81.551) | 81.355% (79.593–81.479) | 87.189% (84.405–87.542) |
| 40% | 133,132 | 81.522% (79.947–82.252) | 81.444% (79.832–82.181) | 87.290% (87.073–88.373) |
| 50% | 111,440 | 81.608% (79.858–83.287) | 81.531% (79.726–83.240) | 87.754% (86.752–89.744) |
| 60% | 88,540 | 81.961% (81.480–83.611) | 81.884% (81.381–83.550) | 88.067% (87.026–90.678) |
| 70% | 66,129 | 81.756% (81.048–83.798) | 81.660% (80.939–83.734) | 88.737% (87.585–91.023) |
| 80% | 43,638 | 81.792% (81.129–82.694) | 81.709% (81.009–82.607) | 88.985% (88.055–89.932) |
| 90% | 22,024 | 82.355% (79.663–85.071) | 82.279% (79.549–85.021) | 88.776% (88.014–92.177) |
Generalization conclusion
- Median exactness on genuinely unseen changed forms moves from 77.296% at 10% training knowledge to 82.279% at 90%, a measured +4.983 percentage-point change.
- Unseen all-form exactness moves from 77.397% at 10% training knowledge to 82.355% at 90%, a measured +4.957 percentage-point change.
- Preservation of unseen already-root forms moves from 84.869% at 10% training knowledge to 88.776% at 90%, a measured +3.907 percentage-point change.
- The evidence establishes within-resource transfer across withheld dictionary families. It does not estimate unrelated domains, misspellings, arbitrary compounds, or external corpora.
The complete ten-level table and split ranges remain in the complete generalization appendix; raw counters and provenance are in active machine-readable snapshot. The frozen methodology defines family-level splitting, unseen-surface leakage control, aggregation, and the limits of the claim.
Stemming quality
Runtime performance and linguistic grouping quality are independent dimensions. This section evaluates language TR_TR using the complete validated stemming-quality result matrix and the canonical linguistic-quality methodology, including its overlapping-group relation, dictionary modes, output policies, metrics, and ranking rules.
Radixor's model was trained from the same lexical resource that defines this benchmark's dictionary-reference relation. The result therefore measures same-resource agreement, not independent external linguistic validity. Third-party adapters receive the same evaluated forms but were not trained by this benchmark.
The tables retain both canonical dictionary-processing modes and every applicable output policy without redefining them on each language page. Download the complete machine-readable result snapshot.
Evaluation scope and key findings
The default model is tr-tr-default, loaded from classpath resource org/egothor/stemmer/models/tr-tr-default/stemmer.gz. The following findings compare only deterministic PRIMARY_OUTPUT rows over identical included groups; candidate policies are reported separately as capability analyses.
- ALL_WORDS:
Radixorranks first by balanced accuracy at 0.991555 among 3 deterministic stemmers. The runner-up isSNOWBALL TURKISH DIRECTat 0.612583, a difference of 0.378972. This rank does not imply leadership in throughput or every secondary metric. - LOWERCASE_GROUPS_ONLY:
Radixorranks first by balanced accuracy at 0.991555 among 3 deterministic stemmers. The runner-up isSNOWBALL TURKISH DIRECTat 0.612583, a difference of 0.378972. This rank does not imply leadership in throughput or every secondary metric.
ALL_WORDS
This mode contains 5 result rows, 3 evaluated stemmers, and 3 output policies. Applied-row and form counts are shown per row because adapters share the language corpus but policy rows remain independently auditable. Where at least two deterministic rows are available, PRIMARY_OUTPUT and ALL_CANDIDATES rankings are ordered by unrounded balanced accuracy, followed by MCC, F1, over-stemming rate, over-stemming count, under-stemming rate, and stemmer. ANY_CANDIDATE has no single rank metric and is listed alphabetically. Balanced accuracy is a navigation metric, not a universally authoritative quality score.
PRIMARY_OUTPUT ranking
| Rank | Stemmer | Balanced accuracy | Over-stemming (OI) | Under-stemming (UI) |
|---|---|---|---|---|
| 1 | Radixor | 0.991555 | 0.000000% | 1.688964% |
| 2 | SNOWBALL TURKISH DIRECT | 0.612583 | 0.000303% | 77.483040% |
| 3 | SNOWBALL TURKISH LUCENE FILTER | 0.612537 | 0.000303% | 77.492390% |
Classification metrics
| Rank | Stemmer | Output policy | Precision | Recall | Specificity | Balanced accuracy | Pairwise accuracy | Error rate |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 1.000000 | 0.983110 | 1.000000 | 0.991555 | 0.999990 | 0.000010 |
| 2 | SNOWBALL TURKISH DIRECT | PRIMARY_OUTPUT | 0.978058 | 0.225170 | 0.999997 | 0.612583 | 0.999533 | 0.000467 |
| 3 | SNOWBALL TURKISH LUCENE FILTER | PRIMARY_OUTPUT | 0.978049 | 0.225076 | 0.999997 | 0.612537 | 0.999533 | 0.000467 |
Pair-relation metrics
| Rank | Stemmer | Output policy | F0.5 | F1 | F2 | Jaccard | Fowlkes–Mallows | MCC |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 0.996576 | 0.991483 | 0.986442 | 0.983110 | 0.991519 | 0.991514 |
| 2 | SNOWBALL TURKISH DIRECT | PRIMARY_OUTPUT | 0.586109 | 0.366064 | 0.266144 | 0.224038 | 0.469286 | 0.469171 |
| 3 | SNOWBALL TURKISH LUCENE FILTER | PRIMARY_OUTPUT | 0.585980 | 0.365939 | 0.266039 | 0.223945 | 0.469186 | 0.469072 |
Raw pair counts
| Rank | Stemmer | Output policy | TP | FP | FN | TN | Over error / possible | Under error / possible |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 14572636 | 0 | 250355 | 24749984637 | 0 / 24749984637 | 250355 / 14822991 |
| 2 | SNOWBALL TURKISH DIRECT | PRIMARY_OUTPUT | 3337687 | 74878 | 11485304 | 24749909759 | 74878 / 24749984637 | 11485304 / 14822991 |
| 3 | SNOWBALL TURKISH LUCENE FILTER | PRIMARY_OUTPUT | 3336301 | 74878 | 11486690 | 24749909759 | 74878 / 24749984637 | 11486690 / 14822991 |
ANY_CANDIDATE oracle bounds
These results are measured, not missing. ANY_CANDIDATE answers two separate optimistic questions for each pair: a gold-related pair avoids under-stemming when the candidate sets intersect, while a gold-negative pair avoids over-stemming when some non-colliding candidate selection exists. The oracle may choose a different candidate for the same word in different pairs. Consequently, these decisions do not form one globally realizable predicted relation or one TP/FP/FN/TN confusion matrix. Balanced accuracy, F-scores, Jaccard, Fowlkes–Mallows, and MCC are therefore mathematically not applicable, rather than unknown.
| Stemmer | Optimistic over-stemming (OI) | Optimistic under-stemming (UI) |
|---|---|---|
| Radixor | 0.000000% | 0.000000% |
Oracle-bound pair counts
| Stemmer | Unavoidable over errors / gold-negative pairs | Unrepairable under errors / gold-related pairs |
|---|---|---|
| Radixor | 0 / 24749984637 | 0 / 14822991 |
ALL_CANDIDATES result (no same-language comparator)
| Rank | Stemmer | Balanced accuracy | Over-stemming (OI) | Under-stemming (UI) |
|---|---|---|---|---|
| n/a | Radixor | 1.000000 | 0.000000% | 0.000000% |
Classification metrics
| Rank | Stemmer | Output policy | Precision | Recall | Specificity | Balanced accuracy | Pairwise accuracy | Error rate |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 0.000000 |
Pair-relation metrics
| Rank | Stemmer | Output policy | F0.5 | F1 | F2 | Jaccard | Fowlkes–Mallows | MCC |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 |
Raw pair counts
| Rank | Stemmer | Output policy | TP | FP | FN | TN | Over error / possible | Under error / possible |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 14822991 | 0 | 0 | 24749984637 | 0 / 24749984637 | 0 / 14822991 |
Multi-output analysis
Alternative candidates are capability analyses, not replacements for the deterministic comparison.
| Stemmer | Under pairs repaired | Best-case over pairs avoided | All-candidate collisions added | Multi-candidate forms | Multi-candidate share | Maximum candidates | Total candidate assignments |
|---|---|---|---|---|---|---|---|
| Radixor | 250355 | 0 | 0 | 1640 | 0.736903% | 3 | 224205 |
LOWERCASE_GROUPS_ONLY
This mode contains 5 result rows, 3 evaluated stemmers, and 3 output policies. Applied-row and form counts are shown per row because adapters share the language corpus but policy rows remain independently auditable. Where at least two deterministic rows are available, PRIMARY_OUTPUT and ALL_CANDIDATES rankings are ordered by unrounded balanced accuracy, followed by MCC, F1, over-stemming rate, over-stemming count, under-stemming rate, and stemmer. ANY_CANDIDATE has no single rank metric and is listed alphabetically. Balanced accuracy is a navigation metric, not a universally authoritative quality score.
PRIMARY_OUTPUT ranking
| Rank | Stemmer | Balanced accuracy | Over-stemming (OI) | Under-stemming (UI) |
|---|---|---|---|---|
| 1 | Radixor | 0.991555 | 0.000000% | 1.688964% |
| 2 | SNOWBALL TURKISH DIRECT | 0.612583 | 0.000303% | 77.483040% |
| 3 | SNOWBALL TURKISH LUCENE FILTER | 0.612537 | 0.000303% | 77.492390% |
Classification metrics
| Rank | Stemmer | Output policy | Precision | Recall | Specificity | Balanced accuracy | Pairwise accuracy | Error rate |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 1.000000 | 0.983110 | 1.000000 | 0.991555 | 0.999990 | 0.000010 |
| 2 | SNOWBALL TURKISH DIRECT | PRIMARY_OUTPUT | 0.978058 | 0.225170 | 0.999997 | 0.612583 | 0.999533 | 0.000467 |
| 3 | SNOWBALL TURKISH LUCENE FILTER | PRIMARY_OUTPUT | 0.978049 | 0.225076 | 0.999997 | 0.612537 | 0.999533 | 0.000467 |
Pair-relation metrics
| Rank | Stemmer | Output policy | F0.5 | F1 | F2 | Jaccard | Fowlkes–Mallows | MCC |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 0.996576 | 0.991483 | 0.986442 | 0.983110 | 0.991519 | 0.991514 |
| 2 | SNOWBALL TURKISH DIRECT | PRIMARY_OUTPUT | 0.586109 | 0.366064 | 0.266144 | 0.224038 | 0.469286 | 0.469171 |
| 3 | SNOWBALL TURKISH LUCENE FILTER | PRIMARY_OUTPUT | 0.585980 | 0.365939 | 0.266039 | 0.223945 | 0.469186 | 0.469072 |
Raw pair counts
| Rank | Stemmer | Output policy | TP | FP | FN | TN | Over error / possible | Under error / possible |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 14572636 | 0 | 250355 | 24749984637 | 0 / 24749984637 | 250355 / 14822991 |
| 2 | SNOWBALL TURKISH DIRECT | PRIMARY_OUTPUT | 3337687 | 74878 | 11485304 | 24749909759 | 74878 / 24749984637 | 11485304 / 14822991 |
| 3 | SNOWBALL TURKISH LUCENE FILTER | PRIMARY_OUTPUT | 3336301 | 74878 | 11486690 | 24749909759 | 74878 / 24749984637 | 11486690 / 14822991 |
ANY_CANDIDATE oracle bounds
These results are measured, not missing. ANY_CANDIDATE answers two separate optimistic questions for each pair: a gold-related pair avoids under-stemming when the candidate sets intersect, while a gold-negative pair avoids over-stemming when some non-colliding candidate selection exists. The oracle may choose a different candidate for the same word in different pairs. Consequently, these decisions do not form one globally realizable predicted relation or one TP/FP/FN/TN confusion matrix. Balanced accuracy, F-scores, Jaccard, Fowlkes–Mallows, and MCC are therefore mathematically not applicable, rather than unknown.
| Stemmer | Optimistic over-stemming (OI) | Optimistic under-stemming (UI) |
|---|---|---|
| Radixor | 0.000000% | 0.000000% |
Oracle-bound pair counts
| Stemmer | Unavoidable over errors / gold-negative pairs | Unrepairable under errors / gold-related pairs |
|---|---|---|
| Radixor | 0 / 24749984637 | 0 / 14822991 |
ALL_CANDIDATES result (no same-language comparator)
| Rank | Stemmer | Balanced accuracy | Over-stemming (OI) | Under-stemming (UI) |
|---|---|---|---|---|
| n/a | Radixor | 1.000000 | 0.000000% | 0.000000% |
Classification metrics
| Rank | Stemmer | Output policy | Precision | Recall | Specificity | Balanced accuracy | Pairwise accuracy | Error rate |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 0.000000 |
Pair-relation metrics
| Rank | Stemmer | Output policy | F0.5 | F1 | F2 | Jaccard | Fowlkes–Mallows | MCC |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 |
Raw pair counts
| Rank | Stemmer | Output policy | TP | FP | FN | TN | Over error / possible | Under error / possible |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 14822991 | 0 | 0 | 24749984637 | 0 / 24749984637 | 0 / 14822991 |
Multi-output analysis
Alternative candidates are capability analyses, not replacements for the deterministic comparison.
| Stemmer | Under pairs repaired | Best-case over pairs avoided | All-candidate collisions added | Multi-candidate forms | Multi-candidate share | Maximum candidates | Total candidate assignments |
|---|---|---|---|---|---|---|---|
| Radixor | 250355 | 0 | 0 | 1640 | 0.736903% | 3 | 224205 |
Method and interpretation boundaries
The linguistic-quality methodology defines the overlapping gold relation, both dictionary-processing modes, all three output policies, confusion counts, formulas, undefined denominators, aggregation, ranking, and inapplicable partition metrics. The candidate-policy reference explains why ANY_CANDIDATE is an oracle-assisted capability bound rather than deterministic runtime behavior.
This page preserves the language-specific raw counts, metrics, candidate distributions, comparison availability, and caveats. The machine-readable CSV remains authoritative for every field.
Provenance
- Authoritative source:
docs/benchmarks/data/stemming-quality-2026-09-13.csv - Source SHA-256:
d41e00160cda44758e806c37d210f2d9b90b1ebad22eebe5ece89a40f4ed9ab0 - Evaluation command:
./gradlew stemmingQuality --no-daemon - Dictionary language:
TR_TR - Processing modes:
ALL_WORDS,LOWERCASE_GROUPS_ONLY - Stemmer versions and transitive artifacts: resolved by the repository's JMH Gradle configuration and
gradle.lockfile - Model ID, version, and SHA-256: recorded in every CSV row
- Run date, core source state, JDK, operating system, and hardware: recorded on the benchmark environment page