Skip to content

Runtime architecture

This page owns runtime architecture: component boundaries, model discovery, construction, reduction, and lookup. Model artifact identity, packaging, aggregate membership, generated resources, and release isolation live in Stemmer Models. Begin with Architecture and Reduction for the two-lane mental model; use Reduction Semantics for the exact equivalence contracts and Compatibility and Guarantees for the supported public boundary.

Java component boundaries

Component Responsibility
Root Radixor core Patch commands, dictionary parser, trie construction/lookup, descriptor and registry APIs, loaders; no language data
Individual model module Immutable source input and license; publishes one independently versioned resource JAR
StemmerModelRegistry Deterministic index/descriptor discovery and selection by model ID or language default
StemmerModelDescriptor Immutable public view of validated runtime identity, format, resource, checksum, and source URL
Model convention plugin Validates inputs and generates the resource namespace, descriptor, index, license, and publication
Standard aggregate POM-only transitive runtime dependencies for 31 standard models
Extended aggregate POM-only transitive runtime dependencies for the other 113 active models
Filtered aggregate POM-only opt-in dependencies for 10 non-default alternatives
Verification classpaths Direct individual-model dependencies for tests, quality evaluation, and JMH, including optional PoliMorf
Models BOM POM-only recommended individual model versions in Maven dependency management
Documentation staging Reviewed docs/, including the checked-in generated catalog; staging independently regenerates a byte-identical catalog under build/mkdocs-source/
Release workflows Independent core, one-model, and catalog publication boundaries

Read Model Selection and Loading for executable application examples and Stemmer Models for artifact maintenance.

Python component boundaries

The Python distribution is a separate native implementation rather than a JVM wrapper. The radixor wheel contains the Rust/PyO3 runtime but no language data. Its mandatory radixor-models-standard dependency supplies 31 validated, precompiled, GZip-compressed version 7 .rxc tries. It does not use Java model JARs, ServiceLoader, descriptors, or the Java registry.

Stemmer("<alias>") resolves and synchronously loads a compiled standard model; it does not parse a textual dictionary at application startup. radixor.compile(...) remains available for application-owned textual dictionaries, and Stemmer(compiled=...) loads the resulting version 7 artifact. The Java, Python, and Python-C in-memory layouts are intentionally different; the shared dictionary syntax and version 7 binary stream are their interoperability boundaries. See Radixor for Python and Compiling Dictionaries in Python.

Runtime model discovery and loading

The implemented sequence is:

  1. use the thread context ClassLoader, or an explicit non-null loader;
  2. enumerate every META-INF/radixor/models.index with ClassLoader.getResources(...);
  3. sort index URLs and validate every descriptor path;
  4. read descriptor resources and required properties;
  5. validate model ID, language, exact resource namespace, checksum syntax, format name, and format version;
  6. sort descriptors by model ID and reject duplicate IDs;
  7. resolve either Language.defaultModelId() or an exact explicit model ID;
  8. open the declared model resource with the descriptor's discovering loader;
  9. compare SHA-256 over the compressed bytes;
  10. decompress GZip and parse UTF-8 Radixor dictionary rows;
  11. build and reduce a FrequencyTrie;
  12. optionally compile stored patch strings into CompiledPatchCommand values for the language-oriented compiled API.

Descriptor discovery verifies resource presence before selection. Byte-level checksum verification happens when the selected model is loaded. The registry never scans arbitrary JAR contents and never selects “the first model for a language.”

Default Polish resolution

StemmerPatchTrieLoader.Language.PL_PL declares pl-pl-unimorph in the enum constructor. A language-oriented load creates a context-loader registry and calls requireDefault(PL_PL). If that ID is absent, loading stops with StemmerModelNotFoundException naming org.egothor:radixor-model-pl-pl-unimorph:<version>.

Explicit PoliMorf resolution

registry.require("pl-pl-polimorf") addresses the alternative directly. It neither changes nor consults the Polish default. Both descriptors may coexist; duplicate declarations of either same ID are rejected.

Version axes

Core, model artifact, catalog, source dictionary, and model-format versions are independent. Their owners and compatibility purposes are defined in Stemmer Models; public upgrade guarantees live in Compatibility and Guarantees. Runtime selection uses the installed descriptor and stable model ID rather than assuming equality among version numbers.

Build topology and generated output

Build topology is a packaging concern rather than a runtime-trie concern. The canonical aggregate-project table records the 31-model standard package, 113-model extended package, 10-model filtered package, 154-model BOM, Gradle output ownership, and the distinction between a Maven dependency BOM and the CycloneDX SBOM. The model-category and lifecycle table preserves the registry, publication, benchmark, and quarantine boundaries for every category.

Build-time model packaging

The runtime consumes a validated descriptor and namespaced dictionary resource; it never reads a module's source-input directory. The lossless packaging sequence, descriptor fields, license-specific requirements, validation tasks, and failure conditions are maintained in Stemmer Models and Create or update a model module.

Release and security boundaries

Core, single-model, and catalog tags have separate publication boundaries. See the canonical release-boundary table for the exact artifacts each tag may and may not publish. License inclusion, strict metadata paths, resource presence, SHA-256 verification, unsupported-format rejection, and duplicate-ID rejection form the model integrity boundary; Trust, Security, and Support places those checks within the wider input and supply-chain threat model.

The central idea

Radixor does not store final stems directly as a large flat lookup table. Instead, it stores patch commands that describe how a word form should be transformed into a canonical stem.

For example, if a dictionary states that running should reduce to run, the final runtime artifact does not need to store a full redundant running -> run output string entry in the simplest possible form. It can store a compact transformation command that expresses how to turn the source form into the target form.

That matters because many words share similar transformation patterns. Once those mappings are organized in a trie and compiled into a canonical structure, the result is much smaller and more reusable than a naive direct-output table.

Trie construction flow

The full build-time flow is:

flowchart TB
    accTitle: Radixor model preparation pipeline
    accDescr: Lexical evidence becomes form-to-root transformations, stored patch commands, a mutable trie, a reduced trie, and an immutable compiled trie used for runtime command selection.

    EVIDENCE["Lexical evidence: grouped forms"] --> TRANSFORM["Form-to-root transformations"]
    TRANSFORM --> COMMANDS["Patch commands: stored values, not final stems"]
    COMMANDS --> MUTABLE["Mutable trie: ranked construction evidence"]
    MUTABLE --> REDUCED["Reduced trie: equivalent behavior"]
    REDUCED --> COMPILED["Compiled trie: immutable runtime structure"]

Preparation derives commands from dictionary evidence, ranks them in a mutable trie, reduces equivalent structure, and freezes an immutable runtime trie. Runtime applies the selected command to the original token.

Each stage has a different purpose.

Dictionary input

The textual dictionary groups known word forms under a canonical stem:

run running runs    ran
connect connected   connecting  connection

The first column is the canonical stem. The following tab-separated columns are known variants.

Patch-command generation

Each variant is converted into a patch command that transforms the variant into the stem.

Conceptually:

running -> <patch> -> run
runs    -> <patch> -> run
ran     -> <patch> -> run

If storeOriginal is enabled, the stem itself is also inserted using a canonical no-op patch.

Mutable trie construction

Those patch-command values are inserted into a mutable trie keyed by the source surface form.

Reduction

Equivalent subtrees are merged into canonical reduced nodes.

Before a selected semantic reduction mode is applied, Radixor also performs uniform-subtree contraction. If every reachable entry below a subtree resolves to the same preferred patch command, that subtree can be represented as an accepting leaf for that command. Runtime lookup can then stop at that leaf even when the input word still has remaining characters.

This is a structural optimization of preferred-result lookup. It reduces trie depth in regions where the remaining suffix cannot change the selected command, while preserving the get() result used by the standard stemmer path. The benchmark tables in docs/benchmarks/ are based on this contracted compiled representation.

Compilation

The reduced structure is frozen into an immutable compiled trie optimized for runtime lookup.

Why a trie is used

A trie is useful because many word forms share structural fragments. Instead of storing each word independently, the trie reuses paths and organizes lookup by character traversal.

A trie node can contain:

  • outgoing edges,
  • one or more ordered values,
  • counts aligned with those values.

This is why the structure can represent both:

  • a single preferred result,
  • multiple competing results for the same key.

Stage 1: Mutable construction

The mutable build-time structure is created by FrequencyTrie.Builder.

This stage is optimized for insertion rather than runtime lookup. As dictionary data is added, the builder accumulates:

  • child edges,
  • local values,
  • local frequencies of those values.

Those frequencies are not incidental metadata. They later influence both result ordering and, depending on reduction mode, the semantic identity of subtrees during reduction.

Why the build-time form is mutable

The builder must be easy to extend and easy to aggregate into. That is the opposite of what a runtime lookup structure needs.

Build-time priorities are:

  • flexibility,
  • accumulation of counts,
  • structural growth.

Runtime priorities are:

  • compactness,
  • immutability,
  • fast lookup.

Radixor therefore keeps construction and runtime representation strictly separate.

What a compiled node contains

After reduction and freezing, the runtime structure uses immutable compiled nodes.

A compiled node stores:

  • char[] edgeLabels
  • child-node references aligned with those labels
  • ordered value arrays
  • aligned count arrays

This array-based form is compact and efficient for lookup.

Runtime lookup model

At runtime, lookup is conceptually simple:

  1. traverse the compiled trie by the input key,
  2. reach the node addressed by that key,
  3. retrieve one or more stored patch commands,
  4. apply the chosen patch command to the original word.

The trie itself does not create the final stem string. It selects the stored transformation command. Runtime code should use CompiledPatchCommand.apply(...) so the serialized command is compiled once and reused.

That separation is architecturally important:

  • the trie is responsible for selection,
  • patch application is responsible for transformation.

get() and getAll()

The runtime API exposes two complementary views of the addressed node.

get()

get() returns the locally preferred value stored at that node.

Preference is deterministic:

  1. higher local frequency wins,
  2. shorter textual representation wins,
  3. lexicographically lower textual representation wins,
  4. stable first-seen order acts as the final tie-breaker.

getAll()

getAll() returns all locally stored values in deterministic ranked order.

This is what allows Radixor to preserve ambiguity explicitly instead of forcing every key into a single answer.

Why multiple results can exist

Some stemming systems discard ambiguity early because they insist on returning exactly one answer.

Radixor does not require that simplification. If multiple plausible patch commands exist for a key, the compiled trie can preserve them and the runtime API can expose them.

That is useful when downstream logic wants to:

  • inspect ambiguity,
  • preserve alternatives for retrieval,
  • apply later ranking or domain-specific selection.

Why compiled artifacts are compact

The final compiled trie can be much smaller than the original dictionary for several reasons working together:

  • patch commands are compact,
  • trie paths reuse shared structure,
  • uniform preferred-command subtrees can be contracted into accepting leaves,
  • reduction merges equivalent subtrees,
  • binary persistence stores the already reduced form,
  • GZip compression is applied on top of the binary format.

This is why a very large dictionary can still produce a manageable deployable runtime artifact.

Why preparation can still use more memory

The compactness of the final artifact should not be confused with the memory usage of preparation.

Before reduction has completed, the mutable build-time structure must exist in memory. For large dictionaries, that temporary preparation cost can be noticeably higher than the size of the final persisted artifact or the loaded compiled trie. PoliMorf is the exceptional current case: two complete test constructions took 23.7 and 23.5 seconds, produced 358,993 canonical nodes, and used a task-specific 6 GiB maximum heap. The process peak does not establish the retained heap of the final trie, which is not currently measured separately.

That is why the preferred operational model is usually:

  • compile offline,
  • persist the compiled artifact,
  • load the finished artifact in runtime services.

Determinism as a design principle

Radixor favors deterministic behavior throughout the pipeline.

This appears in:

  • lowercased dictionary parsing,
  • stable value ordering,
  • sorted child descriptors,
  • canonical reduction signatures,
  • reproducible compiled lookup behavior.

Determinism matters not only for tests, but also for operational trust. It makes stemming behavior explainable and reproducible across builds and environments.

Continue with