Package org.egothor.stemmer

Class StemmerKnowledgeExperiment

java.lang.Object

org.egothor.stemmer.StemmerKnowledgeExperiment

public final class StemmerKnowledgeExperiment
extends Object

Evaluates how stemming quality degrades when the compiled trie is built from only a deterministic subset of the available dictionary knowledge.

The experiment operates on whole dictionary entries. For a chosen knowledge percentage, each parsed dictionary line is deterministically included or excluded from the training subset using a seeded SplittableRandom. The resulting subset is compiled into a FrequencyTrie, while the evaluation is performed against all word forms from the original dictionary.

Two lookup APIs are evaluated:

• FrequencyTrie.get(String) through top-1 accuracy
• FrequencyTrie.getAll(String) through global precision, recall, and F1

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final record	StemmerKnowledgeExperiment.ResultRow	One immutable result row of the knowledge experiment.

Field Summary

Fields

Modifier and Type	Field	Description
static final int	KNOWLEDGE_PERCENT_STEP	Step between adjacent evaluated knowledge percentages.
static final int	MAXIMUM_KNOWLEDGE_PERCENT	Maximum supported knowledge percentage.
static final int	MINIMUM_KNOWLEDGE_PERCENT	Minimum supported knowledge percentage.

Constructor Summary

Constructors

Constructor	Description
StemmerKnowledgeExperiment()	Creates a new experiment harness.

Method Summary

Modifier and Type	Method	Description
List<StemmerKnowledgeExperiment.ResultRow>	evaluate(Reader reader, String sourceDescription, String languageLabel, long seed)	Evaluates a dictionary provided through an arbitrary reader.
List<StemmerKnowledgeExperiment.ResultRow>	evaluateAllBundledLanguages(long seed)	Evaluates all supported bundled dictionaries using the supplied seed.
List<StemmerKnowledgeExperiment.ResultRow>	evaluateBundledLanguage(StemmerPatchTrieLoader.Language language, long seed)	Evaluates one bundled dictionary across all supported experiment configurations.
List<StemmerKnowledgeExperiment.ResultRow>	evaluatePath(Path dictionaryPath, long seed)	Evaluates one filesystem dictionary across all supported experiment configurations.
static void	writeCsv(Path outputPath, List<StemmerKnowledgeExperiment.ResultRow> rows)	Writes result rows as UTF-8 CSV with a stable fixed header.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

MINIMUM_KNOWLEDGE_PERCENT

public static final int MINIMUM_KNOWLEDGE_PERCENT

Minimum supported knowledge percentage.

See Also:

• Constant Field Values

MAXIMUM_KNOWLEDGE_PERCENT

public static final int MAXIMUM_KNOWLEDGE_PERCENT

Maximum supported knowledge percentage.

See Also:

• Constant Field Values

KNOWLEDGE_PERCENT_STEP

public static final int KNOWLEDGE_PERCENT_STEP

Step between adjacent evaluated knowledge percentages.

See Also:

• Constant Field Values

Constructor Details

StemmerKnowledgeExperiment

public StemmerKnowledgeExperiment()

Creates a new experiment harness.

Method Details

evaluateAllBundledLanguages

public List<StemmerKnowledgeExperiment.ResultRow> evaluateAllBundledLanguages(long seed)
                                                                       throws IOException

Evaluates all supported bundled dictionaries using the supplied seed.

Parameters:

seed - deterministic sampling seed

Returns:

immutable ordered list of experiment rows

Throws:

IOException - if reading a bundled dictionary fails

evaluateBundledLanguage

public List<StemmerKnowledgeExperiment.ResultRow> evaluateBundledLanguage(StemmerPatchTrieLoader.Language language,
 long seed)
                                                                   throws IOException

Evaluates one bundled dictionary across all supported experiment configurations.

Parameters:

language - bundled language dictionary

seed - deterministic sampling seed

Returns:

immutable ordered list of experiment rows

Throws:

NullPointerException - if language is null

IOException - if reading the bundled dictionary fails

evaluatePath

public List<StemmerKnowledgeExperiment.ResultRow> evaluatePath(Path dictionaryPath,
 long seed)
                                                        throws IOException

Evaluates one filesystem dictionary across all supported experiment configurations.

Parameters:

dictionaryPath - path to a dictionary file

seed - deterministic sampling seed

Returns:

immutable ordered list of experiment rows

Throws:

NullPointerException - if dictionaryPath is null

IOException - if reading fails

evaluate

public List<StemmerKnowledgeExperiment.ResultRow> evaluate(Reader reader,
 String sourceDescription,
 String languageLabel,
 long seed)
                                                    throws IOException

Evaluates a dictionary provided through an arbitrary reader.

Parameters:

reader - source reader

sourceDescription - logical source description

languageLabel - label stored in the result rows

seed - deterministic sampling seed

Returns:

immutable ordered list of experiment rows

Throws:

NullPointerException - if any argument except seed is null

IOException - if parsing fails

writeCsv

public static void writeCsv(Path outputPath,
 List<StemmerKnowledgeExperiment.ResultRow> rows)
                     throws IOException

Writes result rows as UTF-8 CSV with a stable fixed header.

Parameters:

outputPath - target file path

rows - rows to write

Throws:

NullPointerException - if any argument is null

IOException - if writing fails