Skip to content

Why Radixor is different

Radixor is dictionary-trained, not dictionary-bound.

The source dictionaries are build-time evidence from which Radixor learns word-to-stem transformations. The runtime artifact is not a flat table that can only answer words already present in that evidence. It is a compact, deterministic trie of patch commands.

That distinction is the shortest way to understand the project.

A learned transformation stemmer

A conventional dictionary lookup stores a relationship such as:

running -> run

Radixor instead derives a transformation that can be represented conceptually as:

running -> <patch command> -> run

Many word forms share the same transformation behavior. Radixor organizes those commands in a trie, reduces structurally equivalent regions, contracts uniform preferred-command subtrees, and freezes the result into an immutable compiled runtime structure.

The pipeline is therefore:

flowchart TB
    accTitle: Radixor model preparation pipeline
    accDescr: Lexical evidence becomes form-to-root transformations, stored patch commands, a mutable trie, a reduced trie, and an immutable compiled trie used for runtime command selection.

    EVIDENCE["Lexical evidence: grouped forms"] --> TRANSFORM["Form-to-root transformations"]
    TRANSFORM --> COMMANDS["Patch commands: stored values, not final stems"]
    COMMANDS --> MUTABLE["Mutable trie: ranked construction evidence"]
    MUTABLE --> REDUCED["Reduced trie: equivalent behavior"]
    REDUCED --> COMPILED["Compiled trie: immutable runtime structure"]

In words: preparation derives patch commands from lexical form-to-root evidence, ranks them in a mutable trie, reduces equivalent structure, and freezes the result. Runtime selects a stored command and applies it to the original token; the trie does not store a closed table of final stems.

The dictionary is important because it supplies the linguistic evidence. It does not define a closed runtime vocabulary.

The 2001 patch-command formulation

The patch-command method used by the Egothor/Radixor lineage is documented in Leo Galambos's 2001 publication:

Leo Galambos, “Lemmatizer for Document Information Retrieval Systems in JAVA,” SOFSEM 2001: Theory and Practice of Informatics, Lecture Notes in Computer Science 2234, pp. 243–252, 2001. doi:10.1007/3-540-45627-9_21

The paper formulates a general patch command, or P-command, as a sequence of partial edit commands derived from a minimum-cost path between a word form and its stem. It covers removal, insertion, replacement, and leaving matching text unchanged; gives the commands a compact serialized representation; applies them from the end of the word to reflect predominantly suffixal morphology; and uses the resulting transformations as values in a trie. It is the historical method reference for the patch-command representation in this implementation lineage. Current Radixor extends the implementation with configurable weighted edit costs and traversal, compiled commands, reduction, persistence, and deterministic multi-result semantics.

What a patch command looks like

A serialized patch command is a compact sequence of two-character instructions. The first character is an opcode; the second is either a literal character or a compact count. For counted instructions, a means one, b means two, c means three, and so on.

Instruction Argument Meaning at the current traversal position
D<count> Encoded count Delete source characters.
I<char> Literal character Insert the character into the result.
R<char> Literal character Replace the source character with the argument.
-<count> Encoded count Keep matching source characters unchanged and advance.
Na Canonical fixed argument Make no change; this is Radixor's serialized no-operation command.

The serialized opcodes are related to, but not identical with, the D/I/R/M cost notation used by the benchmarks. In a label such as D2I1R1M0, M is the cost of a matching dynamic-programming transition. When such a match must be represented inside a serialized command, it is emitted as -<count>; Na instead denotes a complete no-operation command. See the edit-cost methodology for the benchmark notation.

The historical and default traversal runs from right to left. Consider the dictionary evidence:

running -> run

Its patch command is:

Dd

D means delete and d encodes four characters. Applied from the right, the instruction removes g, n, i, and n, leaving run. No instruction is needed for the unchanged prefix, so the representation stays compact. More complex commands concatenate instruction pairs, for example combining skips, replacements, deletions, and insertions in their execution order.

For example, a command containing several instruction pairs is:

feet -> foot    -aRoRo

From right to left, -a keeps the final t; the two Ro instructions replace the preceding two e characters with o. The unchanged initial f needs no serialized instruction.

The text command is the portable learned value stored by the trie-building process. At runtime, Radixor compiles it into an immutable specialized command object rather than interpreting the serialized text repeatedly.

What happens to an unseen word?

The compiled trie selects transformation behavior rather than storing a full lemma string for every possible input.

Uniform subtrees can be contracted into accepting leaves. When lookup reaches an accepting leaf whose preferred patch command is already determined, the runtime can apply that command even with input characters remaining. This is one of the ways the compiled model can generalize beyond explicitly observed dictionary forms.

Generalization is not a promise that every arbitrary unknown token has a useful stem. No practical stemmer can make that guarantee. The important property is that Radixor is not limited to exact dictionary membership.

The measured boundary is published rather than implied: the independent 143-language dictionary-family report preserves the frozen 20-language campaign and its separately measured 123-language continuation. Both use five predeclared splits at every 10% knowledge step, while the edit-cost experiment tests how relative command costs and trie structure interact with that transfer. Because the results differ by dictionary, every language page contains its own evidence and bounded conclusions.

For the implementation details, see Architecture.

Why patch commands matter

Patch commands encode how to transform a word rather than merely which string to return.

That gives the runtime model several useful properties:

  • repeated transformation behavior can be shared;
  • trie paths share structural information between related inputs;
  • equivalent subtrees can be reduced;
  • the final command is applied directly to the original token;
  • the runtime can expose a deterministic preferred result;
  • the same compiled node may retain ranked alternative commands when ambiguity should not be discarded.

The result is closer to a compact learned transformation machine than to either a flat dictionary or a handwritten suffix list.

Why the trie matters

The trie is not just a storage container around a dictionary.

It is the structure that makes the learned transformations reusable. Shared paths represent shared input structure; reduction merges equivalent behavior; uniform-subtree contraction can terminate preferred-result lookup early.

This is also why the source dictionary may be very large while the deployed representation remains compact and fast.

Radixor is not three common things

It is not a closed dictionary lemmatizer

A closed dictionary lemmatizer primarily asks whether the current surface form exists in a lexicon or automaton and, if it does, returns stored analyses.

Radixor uses lexical resources differently: it compiles transformation behavior from them.

It is not another fixed suffix-rule stemmer

Porter- and Snowball-family stemmers encode explicit rules for a language. Those systems can generalize because the rules apply to unseen text, but the rules themselves are fixed algorithmic knowledge.

Radixor learns its transformation behavior from language data and then compiles that behavior into its runtime trie.

It is not a full morphological analyzer

A full analyzer may return lemmas, parts of speech, grammatical tags, and multiple analyses. That is valuable when applications need morphological interpretation.

Radixor has a narrower search-oriented objective: produce compact, term conflation with predictable runtime cost. It can preserve multiple stemming candidates, but it does not attempt to become a general-purpose morphological analysis framework.

What is distinctive about the combination

Any one ingredient in isolation is familiar:

  • dictionaries are familiar;
  • tries are familiar;
  • string edit commands are familiar;
  • subtree reduction is familiar.

The distinctive architecture is their combination:

lexical evidence → patch commands → trie organization → semantic reduction → compact deterministic runtime transformation

That architecture separates expensive learning and compilation from hot-path runtime work.

Modern Radixor adds more than the historical implementation

Radixor preserves the useful Egothor idea while rebuilding the operational model for current software:

  • immutable compiled tries;
  • compiled patch-command objects rather than repeated textual interpretation;
  • deterministic ranked multi-result lookup;
  • configurable reduction semantics;
  • uniform-subtree contraction;
  • binary persistence;
  • independent language-model versioning and integrity verification;
  • reopening and extending compiled structures;
  • a complete Java reference runtime plus native Python and Python-C runtimes;
  • reproducible quality and performance benchmark infrastructure.

For the historical lineage and how Stempel, Morfologik, Snowball, and other comparators relate to this architecture, continue with Technology and Lineage.