Indonesian stemmer benchmarks
This page reports dictionary corpus, exact-root agreement, and runtime evidence for the independently available id-id-default Indonesian model.
Dictionary size: . The exact count is 21,296 distinct usable word forms after parser-compatible filtering and exact, case-preserved deduplication. Stars rank dictionary size relative to all benchmarked dictionaries in five nearly equal groups; they do not measure linguistic quality or benchmark accuracy.
The language metadata declares left-to-right writing.
All speed values are environment-specific and were measured on the hardware and JVM listed in the benchmark overview. The command distribution, exact-root accuracy, and speed tables belong to the published 2026-09-11 Radixor/Java 4.4.0 snapshot. Speed benchmark operations process changed tokens. Accuracy uses the complete Radixor dictionary for the language.
How to read this page
Start with the corpus and patch-command distribution, then compare exact-root agreement with runtime. Dictionary-size stars are contextual metadata, not an accuracy result.
Dictionary corpus
| Model ID | Model version | Language | Dictionary rows | Distinct usable forms | Complete quality tokens | Already-root tokens | Changed tokens | Timing workload | JMH timing tokens |
|---|---|---|---|---|---|---|---|---|---|
id-id-default |
1.0.0 |
ID_ID |
3,877 | 21,296 | 21,298 | 3,877 | 17,421 | changed tokens | 17,421 |
Radixor patch-command distribution
Radixor stores the preferred transformation for each normalized dictionary word form as a compiled patch command. This distribution shows which runtime command class is selected by the trained trie for the complete default-model dictionary. The total number of preferred patch commands analyzed for this language is 21,298.
| Command class | Meaning | Word forms | Share |
|---|---|---|---|
BackwardCompoundCommand |
Applies a multi-step backward patch made from skip, delete, insert, and replace operations. | 12,539 | 58.874% |
DeleteSuffixCommand |
Deletes one or more trailing characters from the word form. | 4,880 | 22.913% |
PreserveCommand |
Returns the word form unchanged because it already matches the preferred root. | 3,879 | 18.213% |
Accuracy
Accuracy uses the complete dictionary and reports exact agreement with the dictionary root for each identified model and candidate.
| Stemmer | All exact | Changed exact | Root preserved | Note |
|---|---|---|---|---|
| Radixor | 99.991% | 99.989% | 100.000% | Exact model-ID benchmark; measured in this snapshot. |
| Official Snowball direct (Java) | 81.623% | 79.846% | 89.605% | Official Snowball 3.1.0 generated Java stemmer; measured in this snapshot. |
| Lucene SnowballFilter | 81.623% | 79.846% | 89.605% | Lucene integration of the matching Snowball algorithm; measured in this snapshot. |
Speed
Speed uses JMH average time, 3 warmup iterations, 5 measurement iterations, 3 independent forks, and 1 thread.
The canonical timing workload uses changed tokens when available; a root-only dictionary uses its complete root-preservation corpus. Smaller populations are repeated deterministically to the timing minimum. Relative factors use the Radixor row as the baseline.
| Stemmer | Benchmark method | Score ms/op | Error ms | ns/token | Relative vs Radixor | Note |
|---|---|---|---|---|---|---|
| Radixor | radixor[id-id-default] |
1.602 | 0.212 | 92.0 | 1.000 | Canonical model timing workload; measured in this snapshot. |
| Official Snowball direct (Java) | snowballDirect[INDONESIAN] |
1.888 | 0.211 | 108.4 | 1.178 | Official generated Java stemmer; measured in this snapshot. The reported Radixor and Snowball 99.9% JMH intervals overlap; this summary does not support a directional conclusion, and the runtime ratio remains a point estimate. |
| Lucene SnowballFilter | luceneSnowballFilter[INDONESIAN] |
2.644 | 0.280 | 151.8 | 1.650 | Lucene TokenStream integration; measured in this snapshot. |
Interpretation notes
- The star tier reflects only relative distinct-form count among all benchmarked dictionaries.
- Values shown above come from the identified canonical benchmark snapshot.
- Runtime and exact-root agreement describe different properties and must be interpreted together.
Dictionary-family generalization conclusion
This is the language-specific conclusion from the independent radixor-generalization-v1 baseline
experiment. It is intentionally separate from the wider edit-cost protocol below; values from
the two frozen snapshots are not substituted for one another.
Evidence
Model id-id-default version 1.0.0 is evaluated over five
predeclared nested splits. Unseen metrics remove withheld occurrences whose normalized surface
also appeared in training. Parentheses show the observed split minimum–maximum.
| Training rows | Median unseen occurrences | Unseen all exact | Unseen changed exact | Unseen root preserved |
|---|---|---|---|---|
| 10% | 19,211 | 25.084% (24.160–25.386) | 8.790% (7.665–9.036) | 98.653% (98.424–98.825) |
| 20% | 17,102 | 25.219% (24.846–25.535) | 8.976% (8.465–9.414) | 98.420% (97.840–98.775) |
| 30% | 14,933 | 25.486% (25.119–25.775) | 9.417% (8.774–9.572) | 98.563% (98.047–98.674) |
| 40% | 12,805 | 25.615% (25.342–25.860) | 9.478% (9.047–9.886) | 98.366% (98.108–98.710) |
| 50% | 10,630 | 25.680% (25.254–25.756) | 9.500% (8.838–9.904) | 98.503% (98.246–98.916) |
| 60% | 8,477 | 25.561% (25.455–25.857) | 9.399% (9.005–9.667) | 98.645% (97.742–98.904) |
| 70% | 6,413 | 25.443% (25.035–25.925) | 9.374% (9.002–9.848) | 98.194% (97.762–98.709) |
| 80% | 4,313 | 25.369% (24.948–26.210) | 9.489% (9.065–10.139) | 98.189% (98.065–98.450) |
| 90% | 2,128 | 25.191% (24.658–26.363) | 8.960% (8.724–10.172) | 97.938% (97.680–98.969) |
Generalization conclusion
- Median exactness on genuinely unseen changed forms moves from 8.790% at 10% training knowledge to 8.960% at 90%, a measured +0.170 percentage-point change.
- Unseen all-form exactness moves from 25.084% at 10% training knowledge to 25.191% at 90%, a measured +0.106 percentage-point change.
- Preservation of unseen already-root forms moves from 98.653% at 10% training knowledge to 97.938% at 90%, a measured -0.715 percentage-point change.
- The evidence establishes within-resource transfer across withheld dictionary families. It does not estimate unrelated domains, misspellings, arbitrary compounds, or external corpora.
The complete ten-level table and split ranges remain in the complete generalization appendix; raw counters and provenance are in active machine-readable snapshot. The frozen methodology defines family-level splitting, unseen-surface leakage control, aggregation, and the limits of the claim.
Stemming quality
Runtime performance and linguistic grouping quality are independent dimensions. This section evaluates language ID_ID using the complete validated stemming-quality result matrix and the canonical linguistic-quality methodology, including its overlapping-group relation, dictionary modes, output policies, metrics, and ranking rules.
Radixor's model was trained from the same lexical resource that defines this benchmark's dictionary-reference relation. The result therefore measures same-resource agreement, not independent external linguistic validity. Third-party adapters receive the same evaluated forms but were not trained by this benchmark.
The tables retain both canonical dictionary-processing modes and every applicable output policy without redefining them on each language page. Download the complete machine-readable result snapshot.
Evaluation scope and key findings
The default model is id-id-default, loaded from classpath resource org/egothor/stemmer/models/id-id-default/stemmer.gz. The following findings compare only deterministic PRIMARY_OUTPUT rows over identical included groups; candidate policies are reported separately as capability analyses.
- ALL_WORDS:
Radixorranks first by balanced accuracy at 0.999914 among 4 deterministic stemmers. The runner-up isSNOWBALL INDONESIAN DIRECTat 0.871418, a difference of 0.128497. This rank does not imply leadership in throughput or every secondary metric. - LOWERCASE_GROUPS_ONLY:
Radixorranks first by balanced accuracy at 0.999914 among 4 deterministic stemmers. The runner-up isSNOWBALL INDONESIAN DIRECTat 0.871418, a difference of 0.128497. This rank does not imply leadership in throughput or every secondary metric.
ALL_WORDS
This mode contains 6 result rows, 4 evaluated stemmers, and 3 output policies. Applied-row and form counts are shown per row because adapters share the language corpus but policy rows remain independently auditable. Where at least two deterministic rows are available, PRIMARY_OUTPUT and ALL_CANDIDATES rankings are ordered by unrounded balanced accuracy, followed by MCC, F1, over-stemming rate, over-stemming count, under-stemming rate, and stemmer. ANY_CANDIDATE has no single rank metric and is listed alphabetically. Balanced accuracy is a navigation metric, not a universally authoritative quality score.
PRIMARY_OUTPUT ranking
| Rank | Stemmer | Balanced accuracy | Over-stemming (OI) | Under-stemming (UI) |
|---|---|---|---|---|
| 1 | Radixor | 0.999914 | 0.000000% | 0.017158% |
| 2 | SNOWBALL INDONESIAN DIRECT | 0.871418 | 0.001353% | 25.715126% |
| 3 | SNOWBALL INDONESIAN LUCENE FILTER | 0.871418 | 0.001353% | 25.715126% |
| 4 | INDONESIAN LUCENE INDONESIAN STEM FILTER | 0.861777 | 0.001556% | 27.642964% |
Classification metrics
| Rank | Stemmer | Output policy | Precision | Recall | Specificity | Balanced accuracy | Pairwise accuracy | Error rate |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 1.000000 | 0.999828 | 1.000000 | 0.999914 | 1.000000 | 0.000000 |
| 2 | SNOWBALL INDONESIAN DIRECT | PRIMARY_OUTPUT | 0.951837 | 0.742849 | 0.999986 | 0.871418 | 0.999894 | 0.000106 |
| 3 | SNOWBALL INDONESIAN LUCENE FILTER | PRIMARY_OUTPUT | 0.951837 | 0.742849 | 0.999986 | 0.871418 | 0.999894 | 0.000106 |
| 4 | INDONESIAN LUCENE INDONESIAN STEM FILTER | PRIMARY_OUTPUT | 0.943628 | 0.723570 | 0.999984 | 0.861777 | 0.999885 | 0.000115 |
Pair-relation metrics
| Rank | Stemmer | Output policy | F0.5 | F1 | F2 | Jaccard | Fowlkes–Mallows | MCC |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 0.999966 | 0.999914 | 0.999863 | 0.999828 | 0.999914 | 0.999914 |
| 2 | SNOWBALL INDONESIAN DIRECT | PRIMARY_OUTPUT | 0.901133 | 0.834457 | 0.776967 | 0.715938 | 0.840875 | 0.840826 |
| 3 | SNOWBALL INDONESIAN LUCENE FILTER | PRIMARY_OUTPUT | 0.901133 | 0.834457 | 0.776967 | 0.715938 | 0.840875 | 0.840826 |
| 4 | INDONESIAN LUCENE INDONESIAN STEM FILTER | PRIMARY_OUTPUT | 0.889522 | 0.819076 | 0.758969 | 0.693589 | 0.826306 | 0.826253 |
Raw pair counts
| Rank | Stemmer | Output policy | TP | FP | FN | TN | Over error / possible | Under error / possible |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 81580 | 0 | 14 | 226667566 | 0 / 226667566 | 14 / 81594 |
| 2 | SNOWBALL INDONESIAN DIRECT | PRIMARY_OUTPUT | 60612 | 3067 | 20982 | 226664499 | 3067 / 226667566 | 20982 / 81594 |
| 3 | SNOWBALL INDONESIAN LUCENE FILTER | PRIMARY_OUTPUT | 60612 | 3067 | 20982 | 226664499 | 3067 / 226667566 | 20982 / 81594 |
| 4 | INDONESIAN LUCENE INDONESIAN STEM FILTER | PRIMARY_OUTPUT | 59039 | 3527 | 22555 | 226664039 | 3527 / 226667566 | 22555 / 81594 |
ANY_CANDIDATE oracle bounds
These results are measured, not missing. ANY_CANDIDATE answers two separate optimistic questions for each pair: a gold-related pair avoids under-stemming when the candidate sets intersect, while a gold-negative pair avoids over-stemming when some non-colliding candidate selection exists. The oracle may choose a different candidate for the same word in different pairs. Consequently, these decisions do not form one globally realizable predicted relation or one TP/FP/FN/TN confusion matrix. Balanced accuracy, F-scores, Jaccard, Fowlkes–Mallows, and MCC are therefore mathematically not applicable, rather than unknown.
| Stemmer | Optimistic over-stemming (OI) | Optimistic under-stemming (UI) |
|---|---|---|
| Radixor | 0.000000% | 0.000000% |
Oracle-bound pair counts
| Stemmer | Unavoidable over errors / gold-negative pairs | Unrepairable under errors / gold-related pairs |
|---|---|---|
| Radixor | 0 / 226667566 | 0 / 81594 |
ALL_CANDIDATES result (no same-language comparator)
| Rank | Stemmer | Balanced accuracy | Over-stemming (OI) | Under-stemming (UI) |
|---|---|---|---|---|
| n/a | Radixor | 1.000000 | 0.000000% | 0.000000% |
Classification metrics
| Rank | Stemmer | Output policy | Precision | Recall | Specificity | Balanced accuracy | Pairwise accuracy | Error rate |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 0.000000 |
Pair-relation metrics
| Rank | Stemmer | Output policy | F0.5 | F1 | F2 | Jaccard | Fowlkes–Mallows | MCC |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 |
Raw pair counts
| Rank | Stemmer | Output policy | TP | FP | FN | TN | Over error / possible | Under error / possible |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 81594 | 0 | 0 | 226667566 | 0 / 226667566 | 0 / 81594 |
Multi-output analysis
Alternative candidates are capability analyses, not replacements for the deterministic comparison.
| Stemmer | Under pairs repaired | Best-case over pairs avoided | All-candidate collisions added | Multi-candidate forms | Multi-candidate share | Maximum candidates | Total candidate assignments |
|---|---|---|---|---|---|---|---|
| Radixor | 14 | 0 | 0 | 2 | 0.009391% | 2 | 21298 |
LOWERCASE_GROUPS_ONLY
This mode contains 6 result rows, 4 evaluated stemmers, and 3 output policies. Applied-row and form counts are shown per row because adapters share the language corpus but policy rows remain independently auditable. Where at least two deterministic rows are available, PRIMARY_OUTPUT and ALL_CANDIDATES rankings are ordered by unrounded balanced accuracy, followed by MCC, F1, over-stemming rate, over-stemming count, under-stemming rate, and stemmer. ANY_CANDIDATE has no single rank metric and is listed alphabetically. Balanced accuracy is a navigation metric, not a universally authoritative quality score.
PRIMARY_OUTPUT ranking
| Rank | Stemmer | Balanced accuracy | Over-stemming (OI) | Under-stemming (UI) |
|---|---|---|---|---|
| 1 | Radixor | 0.999914 | 0.000000% | 0.017158% |
| 2 | SNOWBALL INDONESIAN DIRECT | 0.871418 | 0.001353% | 25.715126% |
| 3 | SNOWBALL INDONESIAN LUCENE FILTER | 0.871418 | 0.001353% | 25.715126% |
| 4 | INDONESIAN LUCENE INDONESIAN STEM FILTER | 0.861777 | 0.001556% | 27.642964% |
Classification metrics
| Rank | Stemmer | Output policy | Precision | Recall | Specificity | Balanced accuracy | Pairwise accuracy | Error rate |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 1.000000 | 0.999828 | 1.000000 | 0.999914 | 1.000000 | 0.000000 |
| 2 | SNOWBALL INDONESIAN DIRECT | PRIMARY_OUTPUT | 0.951837 | 0.742849 | 0.999986 | 0.871418 | 0.999894 | 0.000106 |
| 3 | SNOWBALL INDONESIAN LUCENE FILTER | PRIMARY_OUTPUT | 0.951837 | 0.742849 | 0.999986 | 0.871418 | 0.999894 | 0.000106 |
| 4 | INDONESIAN LUCENE INDONESIAN STEM FILTER | PRIMARY_OUTPUT | 0.943628 | 0.723570 | 0.999984 | 0.861777 | 0.999885 | 0.000115 |
Pair-relation metrics
| Rank | Stemmer | Output policy | F0.5 | F1 | F2 | Jaccard | Fowlkes–Mallows | MCC |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 0.999966 | 0.999914 | 0.999863 | 0.999828 | 0.999914 | 0.999914 |
| 2 | SNOWBALL INDONESIAN DIRECT | PRIMARY_OUTPUT | 0.901133 | 0.834457 | 0.776967 | 0.715938 | 0.840875 | 0.840826 |
| 3 | SNOWBALL INDONESIAN LUCENE FILTER | PRIMARY_OUTPUT | 0.901133 | 0.834457 | 0.776967 | 0.715938 | 0.840875 | 0.840826 |
| 4 | INDONESIAN LUCENE INDONESIAN STEM FILTER | PRIMARY_OUTPUT | 0.889522 | 0.819076 | 0.758969 | 0.693589 | 0.826306 | 0.826253 |
Raw pair counts
| Rank | Stemmer | Output policy | TP | FP | FN | TN | Over error / possible | Under error / possible |
|---|---|---|---|---|---|---|---|---|
| 1 | Radixor | PRIMARY_OUTPUT | 81580 | 0 | 14 | 226667566 | 0 / 226667566 | 14 / 81594 |
| 2 | SNOWBALL INDONESIAN DIRECT | PRIMARY_OUTPUT | 60612 | 3067 | 20982 | 226664499 | 3067 / 226667566 | 20982 / 81594 |
| 3 | SNOWBALL INDONESIAN LUCENE FILTER | PRIMARY_OUTPUT | 60612 | 3067 | 20982 | 226664499 | 3067 / 226667566 | 20982 / 81594 |
| 4 | INDONESIAN LUCENE INDONESIAN STEM FILTER | PRIMARY_OUTPUT | 59039 | 3527 | 22555 | 226664039 | 3527 / 226667566 | 22555 / 81594 |
ANY_CANDIDATE oracle bounds
These results are measured, not missing. ANY_CANDIDATE answers two separate optimistic questions for each pair: a gold-related pair avoids under-stemming when the candidate sets intersect, while a gold-negative pair avoids over-stemming when some non-colliding candidate selection exists. The oracle may choose a different candidate for the same word in different pairs. Consequently, these decisions do not form one globally realizable predicted relation or one TP/FP/FN/TN confusion matrix. Balanced accuracy, F-scores, Jaccard, Fowlkes–Mallows, and MCC are therefore mathematically not applicable, rather than unknown.
| Stemmer | Optimistic over-stemming (OI) | Optimistic under-stemming (UI) |
|---|---|---|
| Radixor | 0.000000% | 0.000000% |
Oracle-bound pair counts
| Stemmer | Unavoidable over errors / gold-negative pairs | Unrepairable under errors / gold-related pairs |
|---|---|---|
| Radixor | 0 / 226667566 | 0 / 81594 |
ALL_CANDIDATES result (no same-language comparator)
| Rank | Stemmer | Balanced accuracy | Over-stemming (OI) | Under-stemming (UI) |
|---|---|---|---|---|
| n/a | Radixor | 1.000000 | 0.000000% | 0.000000% |
Classification metrics
| Rank | Stemmer | Output policy | Precision | Recall | Specificity | Balanced accuracy | Pairwise accuracy | Error rate |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 0.000000 |
Pair-relation metrics
| Rank | Stemmer | Output policy | F0.5 | F1 | F2 | Jaccard | Fowlkes–Mallows | MCC |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | 1.000000 |
Raw pair counts
| Rank | Stemmer | Output policy | TP | FP | FN | TN | Over error / possible | Under error / possible |
|---|---|---|---|---|---|---|---|---|
| n/a | Radixor | ALL_CANDIDATES | 81594 | 0 | 0 | 226667566 | 0 / 226667566 | 0 / 81594 |
Multi-output analysis
Alternative candidates are capability analyses, not replacements for the deterministic comparison.
| Stemmer | Under pairs repaired | Best-case over pairs avoided | All-candidate collisions added | Multi-candidate forms | Multi-candidate share | Maximum candidates | Total candidate assignments |
|---|---|---|---|---|---|---|---|
| Radixor | 14 | 0 | 0 | 2 | 0.009391% | 2 | 21298 |
Method and interpretation boundaries
The linguistic-quality methodology defines the overlapping gold relation, both dictionary-processing modes, all three output policies, confusion counts, formulas, undefined denominators, aggregation, ranking, and inapplicable partition metrics. The candidate-policy reference explains why ANY_CANDIDATE is an oracle-assisted capability bound rather than deterministic runtime behavior.
This page preserves the language-specific raw counts, metrics, candidate distributions, comparison availability, and caveats. The machine-readable CSV remains authoritative for every field.
Provenance
- Authoritative source:
docs/benchmarks/data/stemming-quality-2026-09-13.csv - Source SHA-256:
d41e00160cda44758e806c37d210f2d9b90b1ebad22eebe5ece89a40f4ed9ab0 - Evaluation command:
./gradlew stemmingQuality --no-daemon - Dictionary language:
ID_ID - Processing modes:
ALL_WORDS,LOWERCASE_GROUPS_ONLY - Stemmer versions and transitive artifacts: resolved by the repository's JMH Gradle configuration and
gradle.lockfile - Model ID, version, and SHA-256: recorded in every CSV row
- Run date, core source state, JDK, operating system, and hardware: recorded on the benchmark environment page